Welcome to the Inedo Forums! Check out the Forums Guide for help getting started.
If you are experiencing any issues with the forum software, please visit the Contact Form on our website and let us know!
Nuget connection timeout
-
Hello,
we are suffering a lot of timeouts using Proget Nuget feed. I have to restart Proget every 2 hours because our build is hanging in NuGet restore step and Proget is logging that the max connection pool size has been reached.
proget | An error occurred in the web application: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding. proget | An error occurred in the web application: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached. An error occurred in the web application: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached. proget | An error occurred in the web application: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached. proget | fail: Microsoft.AspNetCore.Server.Kestrel[13] proget | Connection id "0HM4OCNCDJEIM", Request id "0HM4OCNCDJEIM:00000002": An unhandled exception was thrown by the application. proget | System.InvalidOperationException: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached. proget | at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection) proget | at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions) proget | at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry) proget | at System.Data.SqlClient.SqlConnection.Open() proget | at Inedo.Data.SqlServerDatabaseContext.CreateConnection() proget | at Inedo.Data.DatabaseContext.ExecuteInternal(String storedProcName, GenericDbParameter[] parameters) proget | at Inedo.Data.SqlServerDatabaseContext.ExecuteInternal(String storedProcName, GenericDbParameter[] parameters) proget | at Inedo.Data.StrongDataReader.Read[TRow](Func`1 getReader, Boolean disposeReader)+MoveNext() proget | at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection) proget | at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source) proget | at Inedo.ProGet.Data.DB.Configuration_GetConfiguration() in C:\InedoAgent\BuildMasterTemp\192.168.44.60\Temp\_E79286\Src\ProGetCoreEx\Data\DB.cs:line 243 proget | at Inedo.ProGet.InternalConfig.ReadConfigValues() in C:\InedoAgent\BuildMasterTemp\192.168.44.60\Temp\_E79286\Src\ProGetCoreEx\Configuration\ProGetConfig.cs:line 200 proget | at Inedo.LazyCached`1.GetValue() proget | at Inedo.ProGet.ProGetConfig.Web.get_CustomErrors() in C:\InedoAgent\BuildMasterTemp\192.168.44.60\Temp\_E79286\Src\ProGetCoreEx\Configuration\ProGetConfig.Web.cs:line 63 proget | at Inedo.ProGet.WebApplication.ProGetHttpModule.Error(HttpApplication app) in C:\InedoAgent\BuildMasterTemp\192.168.44.60\Temp\_E79286\Src\ProGet.WebApplication\ProGetHttpModule.cs:line 392 proget | at Inedo.Web.InedoHttpModule.Inedo.Web.IAhWebModule.ErrorAsync(HttpApplication app) proget | at Inedo.Web.AhWebMiddleware.InvokeAsync(HttpContext context) proget | at Inedo.Web.AhWebMiddleware.InvokeAsync(HttpContext context) proget | at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application) proget | An error occurred in the web application: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached. proget | An error occurred in the web application: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached. proget | An error occurred in the web application: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached. proget | An error occurred in the web application: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached. proget | fail: Microsoft.AspNetCore.Server.Kestrel[13] proget | Connection id "0HM4OCNCDJEI2", Request id "0HM4OCNCDJEI2:00000001": An unhandled exception was thrown by the application. proget | System.InvalidOperationException: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached. proget | at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection) proget | at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions) proget | at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry) proget | at System.Data.SqlClient.SqlConnection.Open() proget | at Inedo.Data.SqlServerDatabaseContext.CreateConnection() proget | at Inedo.Data.DatabaseContext.ExecuteInternal(String storedProcName, GenericDbParameter[] parameters) proget | at Inedo.Data.SqlServerDatabaseContext.ExecuteInternal(String storedProcName, GenericDbParameter[] parameters) proget | at Inedo.Data.StrongDataReader.Read[TRow](Func`1 getReader, Boolean disposeReader)+MoveNext() proget | at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection) proget | at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source) proget | at Inedo.ProGet.Data.DB.Configuration_GetConfiguration() in C:\InedoAgent\BuildMasterTemp\192.168.44.60\Temp\_E79286\Src\ProGetCoreEx\Data\DB.cs:line 243 proget | at Inedo.ProGet.InternalConfig.ReadConfigValues() in C:\InedoAgent\BuildMasterTemp\192.168.44.60\Temp\_E79286\Src\ProGetCoreEx\Configuration\ProGetConfig.cs:line 200 proget | at Inedo.LazyCached`1.GetValue() proget | at Inedo.ProGet.ProGetConfig.Web.get_CustomErrors() in C:\InedoAgent\BuildMasterTemp\192.168.44.60\Temp\_E79286\Src\ProGetCoreEx\Configuration\ProGetConfig.Web.cs:line 63 proget | at Inedo.ProGet.WebApplication.ProGetHttpModule.Error(HttpApplication app) in C:\InedoAgent\BuildMasterTemp\192.168.44.60\Temp\_E79286\Src\ProGet.WebApplication\ProGetHttpModule.cs:line 392 proget | at Inedo.Web.InedoHttpModule.Inedo.Web.IAhWebModule.ErrorAsync(HttpApplication app) proget | at Inedo.Web.AhWebMiddleware.InvokeAsync(HttpContext context) proget | at Inedo.Web.AhWebMiddleware.InvokeAsync(HttpContext context) proget | at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)We are building our software inside docker containers, which means all NuGet packages need to be restored for each build. Sometimes we have 2-3 parallel builds. We are using proget.inedo.com/productimages/inedo/proget:5.3.16 with mcr.microsoft.com/mssql/server:2019-latest.
Any ideas how to overcome this?Best regards
Peter
-
Hi Peter,
Thanks for reposting this; this error is basically what you can expect from a "network stack" being overloaded; each request to a ProGet feed can potentially open another requests to each connector configured (maybe NuGet.org, maybe others), and if you are doing a lot of requests at once, you'll get a lot of network activity queuing up. SQL Server also running on the network, so those just get added to the queue, and eventually you run out of connection.
Do you have a lot of containers running on the same computer that ProGet is running on? If so, that coudl also contribute to the network stack being overloaded.
The best way to solve is with more hardware, or by removing connectors to nuget.org, etc.
Also note that users can overload a server as well, and this is why load balancing will be very important if you want to keep reliability -- see this article to learn more: https://blog.inedo.com/proget-free-to-proget-enterprise
