cannot login on authenticated feeds after upgrade

nuno.guerreiro.rosa_9280

@rhessinger The user actually has view and download. I don't think its related with that since mono works fine. I don't know if me going to the file system and rename all directories/ nupkg to lower case without changing anything in the database could have caused any issue. Is there any data related to package name stored in the DB? That was the only manual operation I did after the upgrade cause of that bug that apparently was found and fixed on the latest version. (proget:5.3.13-ci.2)

Lets say in file system I had My.Package.Hello/My.Package.Hello.nupkg and I renamed to my.package.hello/my.package.hello.nupkg

rhessinger

Hi @nuno-guerreiro-rosa_9280,

If you upload a new package, can you download it then? That would be the best way to verify the casing change did not affect the authentication.

Also, what user directory are you using? Are you using the built-in user directory, Active Directory, or LDAP Legacy?

Thanks,
Rich

nuno.guerreiro.rosa_9280

@rhessinger Built in directory.

rhessinger

Hi @nuno-guerreiro-rosa_9280,

Have you tried uploading a new NuGet package and tried downloading it? I would just like to rule out the directory and file renaming. I'm running the core version locally and I'm not currently able to recreate the authentication issues in my tests.

Thanks,
Rich

nuno.guerreiro.rosa_9280

@rhessinger I will try to upgrade again when I have some time with the latest ci which has the case sensitive fix and let you know the behavior. The mono version is working fine though.

nuno.guerreiro.rosa_9280

@rhessinger I have upgraded to proget:5.3.13-ci.5 and both NuGet restores and docker login started failing

Error
administrator@docker01:/opt/proget$ docker login proget.myinstance.com
Username: example
Password: example
Error response from daemon: login attempt to https://proget.myinstance.com/v2/ failed with status: 401 Unauthorized

/var/log/syslog
Oct 7 08:36:55 docker01 dockerd[1079]: time="2020-10-07T08:36:55.735970715+02:00" level=info msg="Error logging in to v2 endpoint, trying next endpoint: login attempt to https://proget.myinstance.com/v2/ failed with status: 401 Unauthorized"
Oct 7 08:36:55 docker01 dockerd[1079]: time="2020-10-07T08:36:55.736102221+02:00" level=error msg="Handler for POST /v1.37/auth returned error: login attempt to https:/proget.myinstance.com/v2/ failed with status: 401 Unauthorized"

administrator@docker01:/opt/proget$ docker version
Client:
Version: 18.03.1-ce
API version: 1.37
Go version: go1.9.5
Git commit: 9ee9f40
Built: Thu Apr 26 07:17:20 2018
OS/Arch: linux/amd64
Experimental: false
Orchestrator: swarm

Server:
Engine:
Version: 18.03.1-ce
API version: 1.37 (minimum version 1.12)
Go version: go1.9.5
Git commit: 9ee9f40
Built: Thu Apr 26 07:15:30 2018
OS/Arch: linux/amd64

OS information:
NAME="Ubuntu"
VERSION="16.04.6 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.6 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

Dotnet restore on windows fails on all packages:

C:\Program Files\dotnet\sdk\3.1.401\NuGet.targets(128,5): error : Failed to download package 'xxxx.1.0.5' from 'https://proget.myinstance.com/nuget/feed-nuget/v3/flatcontainer/xxxx/1.0.5/xxx.1.0.5.nupkg'.
C:\Program Files\dotnet\sdk\3.1.401\NuGet.targets(128,5): error : Response status code does not indicate success: 401 (Unauthorized).

rhessinger

Hi @nuno-guerreiro-rosa_9280,

I was finally able to recreate the issue. We created a ticket, ILIB-98, to track the issue. It will be released on Friday with ProGet 5.3.13.

Thanks,
Rich

nuno.guerreiro.rosa_9280

@rhessinger Nice to hear, looking forward to try a final version so I can upgrade to .net core version and with some luck, will achieve better performance because I am running it in 4 vCPU server with 32 gb ram and its not very normal that with a single client when doing a net restore it starts throwing timeout exceptions. Have you tried to do some load tests on restore to find out why it overwhelms the server so easily? Let's say I have 40 nuget packages from public nuget.org and 2 from proget. It will still query 40 packages in proget to find out it doesn't exist in the server. I believe those 40 package checks might be overwhelming the server. Just a guess.

rhessinger

Hi @nuno-guerreiro-rosa_9280,

The .NET Core version is definitely more effecient than the mono version. One of the biggest issues with mono is how it manages web requests. Mono reimplemented its web client and connection pool from scratch due to the complexity and internal dependencies of the .NET Framework. This is probably part of the reason you see the mono version tax the server a bit. As for NuGet querying all 40 packages in ProGet, we don't have much control over that. NuGet queries all sources in the list until it finds one. It will do custom sources first then fallback to NuGet.org last. There is an interesting ticket on NuGet's GitHub page discussing the priority (https://github.com/NuGet/Home/issues/3676).

Thanks,
Rich

nuno.guerreiro.rosa_9280

@rhessinger I have upgraded to ProGet v5.3.13 and the login issue still occurs. I had to revert back to progetmono to be able to perform docker login.

nuno.guerreiro.rosa_9280

@rhessinger I have done some more testing and confirm that the NuGet restore is now working however the docker login (Unauthorized error) is still failing so that still remains to be fixed. Maybe the same issue but now with docker login ?

Login did not succeed, error: Error response from daemon: login attempt to https://proget.myinstance.io/v2/ failed with status: 401 Unauthorized

/var/log/syslog

Oct 10 23:54:21 docker01 dockerd[1079]: time="2020-10-10T23:54:21.726102349+02:00" level=info msg="Error logging in to v2 endpoint, trying next endpoint: login attempt to https://proget.myinstance.io/v2/ failed with status: 401 Unauthorized"
Oct 10 23:54:21 docker01 dockerd[1079]: time="2020-10-10T23:54:21.727052689+02:00" level=error msg="Handler for POST /v1.37/auth returned error: login attempt to https://proget.myinstance.io/v2/ failed with status: 401 Unauthorized"

rhessinger

Hi @nuno-guerreiro-rosa_9280,

Thank you for all the extra information. I believe I have found the culprit, but I will need to work with the team internally to determine the proper fix. It looks like the call to /v2/ that tells docker it needs to authenticate is not returning the proper headers. This is a .NET Core only issue.

The reason that Docker is the only feed affected now is because of how the docker client/api works. It basically walks through the process of an API looking for the API to return very specific errors and headers. That is how the client knows what to do next. In the case of auth, it will first hit /v2/ and look for a 401 unauthroized and some specific headers to tell it to redirect to /v2/_auth and to pass an Authorize header. That is not happening. I will update when we have scheduled this for release.

Thanks,
Rich

rhessinger

Hi @nuno-guerreiro-rosa_9280,

The docker login issue has been scheduled to release in ProGet 5.3.14 on October 30, 2020. I will let you know if anything changes.

Thanks,
Rich

nuno.guerreiro.rosa_9280

@rhessinger Hi, Is it possible to release this fix before 30th October or not really ? I am unable to upgrade until then. I guess anyone who uses docker registry authentication in this case.

rhessinger

Hi @nuno-guerreiro-rosa_9280,

Sorry for the late notice but the release of 5.3.14 was pushed back to later today. I'm sorry if this caused inconvieninces for you.

Thanks,
Rich

nuno.guerreiro.rosa_9280

Thank you. this situation seems to be sorted in 5.3.15.