docker login failed via https reverse proxy

viceice

If i put a reverse proxy for https termination infront of proget, docker client can't login because a wrong www-authenticate header. the response tries to redirect to http instead of https which isn't allowed

passed proxy headers (fetched via containous/whoami docker image)

Hostname: 68ff07e4f3c9
IP: 127.0.0.1
IP: 172.22.0.4
RemoteAddr: 172.22.0.2:60372
GET / HTTP/1.1
Host: wai-test.kriese.eu
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9,de-DE;q=0.8,de;q=0.7
Cache-Control: max-age=0
Dnt: 1
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1
X-Forwarded-For: 172.22.0.1
X-Forwarded-Host: wai-test.kriese.eu
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: ee3e7e9207a9
X-Real-Ip: 172.22.0.1

proget response header

curl -sv https://proget-test.kriese.eu/v2/
*   Trying 192.168.158.122:443...
* TCP_NODELAY set
* Connected to proget-test.kriese.eu (192.168.158.122) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=kriese.eu
*  start date: Sep 13 03:22:32 2020 GMT
*  expire date: Dec 12 03:22:32 2020 GMT
*  subjectAltName: host "proget-test.kriese.eu" matched cert's "*.kriese.eu"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x564ea06937c0)
> GET /v2/ HTTP/2
> Host: proget-test.kriese.eu
> user-agent: curl/7.68.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 401
< cache-control: private
< content-type: application/json
< date: Thu, 29 Oct 2020 15:17:28 GMT
< docker-distribution-api-version: registry/2.0
< server: Kestrel
< vary: Authorization
< www-authenticate: Bearer realm="http://proget-test.kriese.eu/v2/_auth",service="proget-test.kriese.eu"
< x-proget-version: 5.3.15.2
< content-length: 145
<
* Connection #0 to host proget-test.kriese.eu left intact
{"errors":[{"code":"UNAUTHORIZED","message":"Anonymous is not permitted to perform the Feeds_ViewFeed task for the current scope.","detail":[]}]}

rhessinger

Hello @viceice,

What version of ProGet are you running? There was a bug that was fixed on ProGet 5.3.15 that corrected the headers in the docker login process when using the proget image. Also, please make sure that you set the Web.BaseUrl to the https reverse proxy URL in the Administration -> Advanced Settings section.

Thanks,
Rich

viceice

I'm using proget.inedo.com/productimages/inedo/proget:5.3.15

If i set Web.BaseUrl, i get license errors, because of loopback nuget v3 connector :-(

rhessinger

Hi @viceice,

I was just about to link you to another forums post about X-Forward-* headers, but I realize that post was with you. Just out of curiosity. Are you using a different URL for your docker feed than your nuget feed? Also in your docker client, do you have your URL registered as an insecure registry?

Thanks,
Rich

viceice

I've created a new minimal test setup to reproduce the login issue (docker desktop on windows 10).

login to to localhost:8082 works
login to to localhost:8083 works
login to to proget-test.kriese.eu fails, because docker tries to login with http instead of https which failes, because my proget does not listen to http://proget-test.kriese.eu

Here is the compose file i use for testing:

version: "2.4"
services:
  proget:
    image: proget.inedo.com/productimages/inedo/proget:5.3.15
    restart: unless-stopped
    container_name: proget
    environment:
      SQL_CONNECTION_STRING: Data Source=mssql; Initial Catalog=proget; User ID=proget; Password=proget
    volumes:
      - proget-packages:/var/proget/packages
    ports:
      - 8082:80/tcp
    networks:
      - mssql
      - default
    labels:
      - "traefik.http.routers.proget.rule=Host(`proget-test.kriese.eu`) || Host(`localhost`)"
      - traefik.enable=true
      - traefik.http.services.proget.loadbalancer.server.port=80
      - traefik.docker.network=proget

  whoami:
    image: containous/whoami
    restart: unless-stopped
    labels:
      - "traefik.http.routers.whoami.rule=Host(`wai-test.kriese.eu`)"
      - traefik.enable=true
      - traefik.docker.network=proget

  proxy:
    image: traefik:2.3.2
    restart: unless-stopped
    command:
      - --entryPoints.web.address=:80
      - --entryPoints.websecure.address=:443
      - --api.insecure=true
      - --providers.docker.exposedByDefault=false
      - --providers.file.directory=/etc/traefik.d
      - --providers.file.watch=true
      - --accesslog=true
      - --accesslog.format=json
      - --accesslog.fields.defaultmode=keep
      - --accesslog.fields.headers.names.RequestAddr=keep
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./conf:/etc/traefik.d:ro
      - ./letsencrypt:/etc/letsencrypt:ro
    ports:
      - 443:443/tcp
      - 8083:80/tcp
      - 8084:8080/tcp

volumes:
  proget-packages:

networks:
  default:
    name: proget
  mssql:
    external: true

and the traefik tls config file ./conf/tls.yml

tls:
  certificates:
    - certFile: /etc/letsencrypt/live/kriese.eu/fullchain.pem
      keyFile: /etc/letsencrypt/live/kriese.eu/privkey.pem

rhessinger

Hi @viceice,

What about your docker daemon.json? By default, docker will use https, unless if the host is listed within the insecure-registries in your docker daemon.json. Can you please verify that URL does not exist in that list?

Thanks,
Rich

viceice

@viceice said in docker login failed via https reverse proxy:

www-authenticate: Bearer realm="http://proget-test.kriese.eu/v2/_auth",service="proget-test.kriese.eu"

docker login proget-test.kriese.eu make a get request to http://proget-test.kriese.eu/v2/ which return the header www-authenticate: Bearer realm="http://proget-test.kriese.eu/v2/_auth",service="proget-test.kriese.eu.
Then docker is maoking a reuest to http://proget-test.kriese.eu/v2/_auth as told in the response, which fails, because protokoll changed from https to http.

The proget auth header should be like:
www-authenticate: Bearer realm="https://proget-test.kriese.eu/v2/_auth",service="proget-test.kriese.eu

The http://proget-test.kriese.eu/v2/_auth is not available, as i do not forward these insecure http requests from proxy to proget

for reference my daemon.json

{
  "registry-mirrors": [],
  "insecure-registries": [],
  "debug": true,
  "experimental": false
}

pretty default config

rhessinger

Hi @viceice,

This is very weird to me. When you call docker login, you said it is initially doing a get to http://proget-test.kriese.eu/v2/ not https://proget-test.kriese.eu/v2/. This means that there is something configured in your docker client to not use https in docker requests. The reason ProGet is returning an http:// based realm is because the initial request is being made non-https URL.

Can you send a screenshot of your command output in the docker login process?

Thanks,
Rich

viceice

Hi @rhessinger,

Sorry my bad, it's a typo, first request is going to https://proget-test.kriese.eu/v2/ and second is going to http://proget-test.kriese.eu/v2/_auth

The second request is expected by docker to go to http because of the wrong www-authenticate header

atripp

Hi @viceice

The http://proget-test.kriese.eu/ being returned as the Realm because it's set as the BaseUrl.

If you remove that, then it will work; however, it would cause your instance of ProGet Free to report licensing errors due to your reverse proxy configuration.

Ultimately this is a reverse proxy issue, and we might be able to have the Relm respect the X-Forward* headers.... however, I'm not sure if that's appropriate / okay to do?

It might be a security problem? Can you find any documentation or discussion on this topic?

The code change is easy, but we want to confirm it's okay to make before considering it further.

Cheers,
Alana

viceice

Hi @atripp

My BaseUrl is empty, so proget is correctly assuming the hostname from headers.
But as you said, it isn't respecting the X-Forward* headers.
That why proget falsy returns a http url, because the reverse proxy is doing http to proget.

If i add the baseUrl, i'll get licensing issues for a nuget v3 loopback connector, so i was told to empty baseUrl. see https://forums.inedo.com/topic/3037/how-to-configure-the-proget-free-with-self-connector/13

So my assumption would be, that proget will respect the X-Forward* headers as already doing in other feeds.

Another options would be, if we can configure proget to listen on https with a provided tls certificate, so i would have end-to-end tls encryption.

rhessinger

Hi @viceice,

Looking back at your other forums post. The last fix we released should allow you to set the Web.BaseUrl and with the addition of X-Forward* headers in your reverse proxy, it should not give you those license errors any more? Were you able to verify that it still does give oyu the licenses errors?

Thanks,
Rich

viceice

ok, setting BaseUrl fixes the docker login but causes new license violations

I now have 2 feeds a and b and configured a loopback connector on a to feed b

Errors / Warnings:

The visualon.test test nuget packages was added to feed b

btw: license issues are only go away, if i configure a diffrent license key. If i simply try to update existing key, license violations aren't removed. So the docu is wrong.

I think the solution would be that the licensing code checks passed request url against the configured baseUrl and ignore those requests.

rhessinger

Hi @viceice,

Thanks for testing this for me. I think I may see your issue. When the Web.BaseUrl is set to https://proget-test.kriese.eu, the connector should use the URL https://proget-test.kriese.eu/nuget/b/v3/inedx.json. If you set it up that way, do you still get the license violations?

I'll look into the license key issue. It should work as the documentation describes it.

Thanks,
Rich

viceice

@rhessinger said in docker login failed via https reverse proxy:

Thanks for testing this for me. I think I may see your issue. When the Web.BaseUrl is set to https://proget-test.kriese.eu, the connector should use the URL https://proget-test.kriese.eu/nuget/b/v3/inedx.json. If you set it up that way, do you still get the license violations?

yes, i can use https://proget-test.kriese.eu/nuget/b/v3/index.json or http://localhost/nuget/b/v3/index.json and always get licensing issues. (baseUrl=https://proget-test.kriese.eu)

So i'll set the BaseUrl on our other proget too and get starting licensing issues, even if the connector url matches the BaseUrl.

One thing to note is, if i set the BaseUrl to https://proget-test.kriese.eu all requests from proget connector will go through the proxy. So i think this is causing the licensing issue.

atripp

@viceice said in docker login failed via https reverse proxy:

So my assumption would be, that proget will respect the X-Forward* headers as already doing in other feeds.

You're right, when writing the Basic realm="<url>" header, ProGet does not look at the X-Forward* headers. I'm not sure if we should? Would that be a security problem? It kind of seems like it might? Or maybe not? I didn't find anything about the topic discussed on the 'net after a quick search...

How about just configuring your reverse proxy to just rewrite that realm in the auth header to go from http to https? I suspect, thatw ould help.

If you can do that, and then we can update the docs on how to do it, other Free users would be very much appreciative :)

viceice

As for the security concern: maybe add an environment config option to pass trusted proxy ips like traefik supports forwardedHeaders.trustedIPs

Maybe another option is to use the ASP.NET Core proxy config options to solve the issue.

Ok, my reverse proxy isn't able to rewrite headers traefik/traefik#6047

So i will go with setting BaseUrl and hope that the licensing issues get fixed in a later version.
I only see the issues when using an admin user, which we normally not use.
I also hope you won't block this usage scenario in a future version whithout a proper solution.

rhessinger

Hi @viceice,

I did a few searches and have you tried setting the hostsProxyHeaders> Maybe something like this: traefik.frontend.headers.hostsProxyHeaders=www-authenticate? The docs make it sound like it will look in those headers for URLs that are proxied. You could give that a shot.

Thanks,
Rich