Issue with IIS AppPool / Connection Pool Timeout after upgrading to 4.0.5

Update This has been fixed in ProGet 4.0.6

Hi,
I just upgraded to Proget 4.0.5 and I now have periodic issue with it. At some point it throws the following error and it is only recoverable with an IIS app-pool recycle (IIS have yellow sceen of death showing the same error and nuget Api is unreachable)...

Image Text

The underlying error appears to be:

Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.

I don't know what exactly is causing this but it seems to be proportional to the usage of the website...

Product: ProGet
Version: 4.0.5

Same issue with my proget instance after upgrade.
Spent 2 hours migrating feeds to new format and had to rollback everything.

We are currently investigating this; any perfmon data will help, such as an abnormally high # of connections. Even identifying which "simulatenous" request number it fails on is a big help.

We're unable to reproduce this in our test labs unless we drastically lower the number of IIS worker threads or SQL connection pool size, but we believe it's related to v4 async optimizations made for the extremely high volume of requests that come in from NuGet.exe and npm.

As a work-around you can increase the number of connections available in your sql connection pool.

This appears to be caused by an corner-case bug with using SQLClient/async libraries that cause a deadlock while trying to retrieve from a connection pool; we are currently submitting a bug report to Microsoft, but will work-around it in our code.

We will be ship a new ProGet version in a maintenance release tomorrow.

There is quite a lot of other errors that are logged that may or may not be related:
Image Text

and:
Image Text

This seems to happen for every depended-on package gotten through the nuget connector (packages on the official feed):
Image Text

Regarding the abnormally high # of connection, I was able to have the issue with only 2 person connected. So high load/traffic is not the issue. I'll trace the SQL server tomorrow to check if there is something weird happening...

Thanks again,

Carl.

It's hard to say if the errors are related; once the deadlock occurs, it seems to put the .NET runtime in a really interfrastic state, and "impossible" things keep happening.

Because it's an error internal to the .NET/SQL client connection, we can't tell exactly what conditions cause it to occur.

However, in our test lab, after significantly lowering the connection pool size, we can consistently reproduce this behavior. After we tried our work-around (de-async connection creation), the issue went away immediately. After overnight tests, we haven't seen it again.

A fix will be shipped soon.

Update! This has been fixed in ProGet 4.0.6