We are getting connection pool timeout

[sandy.wang]

Hi,

We have been running Proget 4.8.5 since last Oct.
Recently we started getting error like:

Logged: 6/18/2018 4:28:56 PM
Level: Error
Category: Web
Message: An error occurred processing a GET request to http://integrity-nuget.twilight.azure/nuget/integrity-production/FindPackagesById()?id='FoooBarr': Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
Details: System.InvalidOperationException: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
at System.Data.Common.ADP.ExceptionWithStackTrace(Exception e)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Inedo.Data.SqlServerDatabaseContext.<CreateConnectionAsync>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Inedo.LazyAsync1.<GetValueAsync>d__12.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Inedo.Data.DatabaseContext.<ExecuteInternalAsync>d__33.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Inedo.Data.DatabaseContext.<ExecuteTableAsync>d__321.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Inedo.ProGet.Feeds.NuGet.LocalPackageSource.<FindPackagesByIdAsync>d__15.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Inedo.ProGet.Feeds.NuGet.NuGetFeed.<FindPackagesByIdAsync>d__27.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Inedo.ProGet.WebApplication.FeedEndpoints.NuGet.ODataHandler.<ProcessVirtualRequestAsync>d__10.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Inedo.ProGet.WebApplication.FeedEndpoints.NuGet.ODataHandler.<ProcessRequestAsync>d__9.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Inedo.ProGet.WebApplication.FeedEndpoints.FeedEndpointHandler.FeedRequestHandler.<ProcessRequestAsync>d__8.MoveNext()
Close

On our setup, we are using default value for max pool size - 100
Our DBA helped us to check how many connection pool are running around the time the problem occurred, we saw it's over 90.
DBA suspect that the application is not closing and disposing of connections properly.

I googled and found out the same error has been fixed in 4.0.6, but I don't know if the reasons are the same, hard to tell.
https://inedo.com/support/questions/4270

Product: ProGet
Version: 4.8.5

[Alex P]

The issue that you linked has since long been resolved; it was a problem with our beta v4. At that time, the problem was the ProGet was "leaking" some connections (i.e. not properly disposing, as your DBA noted).

So, if you're seeing the problem now, it's unfortunately not so "simple" of a problem to fix. It basically means that there is some sort of latency between the web/database servers and/or the database is not able to respond to queries as fast as the web server is making them.

There's a lot of things that can cause these, from fragmented indexes to low resources on the database server. Do you know what changed recently? Etc. I'm afraid increasing the connection pool likely won't help :/