Downloading NuGet packages parallely causes serious interference

ghollosy_9163

By migrating to ProGet to use as the NuGet server company wide, (as more and more developers switched to the new NuGet server) we were getting strange build errors relating to references coming from NuGet packages.
First, we tried to handle the issue by clearing all local NuGet caches, upgrading/downgrading nuget.exe, but later, it turned out that the source of the problem is, that some of the NuGet package folders in local cache do not contain the correct NuGet package, e.g. folder A.B.C/1.2.3 contains the content of X.Y.Z-1.1.0.nupkg?!?
At the end, we wrote a simple test application to download random .nupkg files parallely from ProGet server and verify the retrieved content of each .nupkg files, the result was disastrous: by trying to download 1000 random .nupkg parallely, only 89 packages were correct, the rest, 911 packages contained .nupkg of an other package being requested at the same time!
Obviously, there is a serious concurreny issue inside ProGet server, that by requesting A.B.C.nupkg, then X.Y.Z.nupkg is received.

This issue was first detected in an environment as:
OS = CentOS, ProGet = 5.2.7, Docker = 19.03.0, DB = MSSQL 2016 SP2, NuGet packages = 350 pieces and ~25K versions alltogether
And confirmed in following environment:
OS = CentOS, ProGet = 5.2.14, Docker = 19.03.0, DB = MSSQL 2016 SP2, NuGet packages = 350 pieces and ~25K versions alltogether
OS = Win10, ProGet = 5.2.14, Docker = 19.03.4, DB = MSSQL 2017 Docker, NuGet packages = 1K
OS = Win10, ProGet = 5.2.14 (both self-hosted and IIS hosted), DB = MSSQL 2017 Docker, NuGet packages = 1K

atripp

Hi there;

ProGet is quite optimized as a far as software goes (check out ProGet Case Studies to see the scale that enterprises use it at), and a lot of our users have switched from competitive tools for massive performance gains that our tools have. But you need to put it on proper hardware.

Keep in mind that NuGet tools were designed to operate against NuGet.org, which is run on a massive server farm that sends primarily static content. You are making 1000's of requests, and each request to ProGet is comparatively expensive, because it proxies those requests to NuGet.org (assuming you have a connector loaded), makes database connections, it checks permissions, vulnerabilities, etc.

Each request to the server can result in several subsequent network requests... and it sounds like your ProGet server is a Win10 desktop... there's just no way it's going to keep up with developers hammering it with more powerful workstations.

Check out ProGet Free to ProGet Enterprise to see the performance recommendations we have, and other reasons organizations upgrade.

ghollosy_9163

Thanks for your response, since this issue was reported, we were trying many different test scenarios, environments, but before going to the details, let me highlight the aspect of this issue to avoid further misunderstandings:

The production ProGet Free 5.2.14 server is running in a docker on a Linux (CentOS) host and not on Windows host. Windows OS was used as one of our test environment to check whether this HTTP request/response interference issue is related only to Linux host.
The issue we're having it has nothing to do with how powerful is the host's hardware. Even if our Linux host is "slow", this interference issue should never ever happen, meaning, that by sending an HTTP request to get A.nupkg, in the HTTP response, B.nupkg is received
According to the linked performance recommendations, Free version should be completely enough for small companies (6 developers:) ) as we are.

By performing many-many tests, it turned out that to reproduce the issue the followings required:
A) Self-hosted ProGet server (we used docker on both Linux and Win OS hosts)
B) Windows 10 version 1903(!) as a client
C) Downloading .nupkg files parallelly (3-4 threads are enough) with HTTP connection Keep-Alive

What happens on a problematic environment is that -due to Keep-Alive switch- the client (written in .Net) -after a few requests- starts to reusing TCP connections and sends 3-4 HTTP requests at once through the same TCP channel (it does not wait the response before sending the next request) and when the responses are received it has no idea which HTTP response belongs to which HTTP requests (we used Wireshark to monitor network traffic, and even Wireshark could not point the corresponding HTTP response to a request) and as a result for requesting A.nupkg e.g. B.nupkg is received without any error reported on client side.
NuGet.exe is also affected with this issue, but it happens much rarely then via the test app we wrote.

The issue does not occur if any of the 3 points above are not satisfied:

Client's OS version is earlier, e.g. Windows 10 version 1803 or
Packages are downloaded parallelly, but Keep-Alive is not turned on or
ProGet is hosted in IIS, or a proxy is used in front of ProGet server
Fiddler (or similar tool) is used to examine client-server traffic

It is quite weird why the client can behave differently if the server (ProGet) is self-hosted?!

As a workaround to the current situation, we started to use a proxy jwilder/nginx-proxy in front of ProGet server and now all the developers can download NuGet packages without any interference issue.

atripp

Thanks for the detailed information, that's very interesting and a great assessment. We wouldn't have found this using all of these variables, especially the different Windows desktop version

I suspect this might be a bug in the underlying libraries? We just use an HttpListener and a HttpWorkerRequest, both which should handle all of this, in the simplest way possible. I'll ask around and see if anyone knows.

But I guess, at least you found a work-around which I'm glad to hear!