Lots of "The operation has timed out." errors when ProGet tries to read other feeds from same server?

[c.schuette_7781]

I have a NuGet "default" feed that is connected to the standard nuget.org feed and also includes connectors to three "local" NuGet feeds. I get lots of "The operation has timed out." errors when the build server tries to restore packages:

The callstack is always the same:

Request URL: https://nuget.local/nuget/internal/FindPackagesById()?id=%27Microsoft.Win32.SystemEvents%27&semVerLevel=2.0.0

System.Net.WebException: The operation has timed out.
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location ---
at Inedo.ProGet.Feeds.Connector.GetDirectResponseAsync(String url)
at Inedo.ProGet.Feeds.Connector.GetCachedResponseAsync(String url)
at Inedo.ProGet.Feeds.Connector.GetCachedTextReaderAsync(String url, Boolean bufferEntireResponse)
at Inedo.ProGet.Feeds.NuGet.V2.NuGetFeedClient.PerformRequestWithAbsoluteUrlAsync(String fullUrl)

Looks like there are some "deadlock" issues when ProGet tries to read from feeds that are also handled by the same ProGet instance?

[c.schuette_7781]

This also seems to happen when the build server accesses NuGet and npm feeds on the same ProGet instance at the same time. NPM feed access also causes high CPU load. So maybe the NPM stuff causes this indirectly....?

[stevedennis]

Hi @c-schuette_7781,

I have a NuGet "default" feed that is connected to the standard nuget.org feed and also includes connectors to three "local" NuGet feeds. I

So basically, you're doing a Denial of Service attack against your ProGet server ;)

When the NuGet client makes that FindPackagesById() request, ProGet needs to now make four separate web requests (nuget.org plus the three other feeds). Considering that NuGet client makes 100's of simultaneous requests for different packages, you're going to run into errors like this. Especially with multiple builds (multiple sets of 100's of requests / second).

If you want to handle this level of traffic, you need to use load balancing.

Otherwise, you need to reduce traffic. Switch to the NuGet v3 API, use connector metadata caching, reduce the number of connectors, set a Web.ConcurrentRequestLimit in the admin > advanced, etc.

[c.schuette_7781]

Thank you.

The problem seems to be caused by Web.ConcurrentRequestLimit setting. If the server receives lots of requests and needs to query itself, these requests will be queued when the limit is reached. So they never finish in time and the original request throws the timeout exception.