Welcome to the Inedo Forums! Check out the Forums Guide for help getting started.
If you are experiencing any issues with the forum software, please visit the Contact Form on our website and let us know!
Nuget connection timeout
-
Hello,
we are suffering a lot of timeouts using Proget Nuget feed. I have to restart Proget every 2 hours because our build is hanging in NuGet restore step and Proget is logging that the max connection pool size has been reached.
proget | An error occurred in the web application: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding. proget | An error occurred in the web application: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached. An error occurred in the web application: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached. proget | An error occurred in the web application: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached. proget | fail: Microsoft.AspNetCore.Server.Kestrel[13] proget | Connection id "0HM4OCNCDJEIM", Request id "0HM4OCNCDJEIM:00000002": An unhandled exception was thrown by the application. proget | System.InvalidOperationException: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached. proget | at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection) proget | at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions) proget | at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry) proget | at System.Data.SqlClient.SqlConnection.Open() proget | at Inedo.Data.SqlServerDatabaseContext.CreateConnection() proget | at Inedo.Data.DatabaseContext.ExecuteInternal(String storedProcName, GenericDbParameter[] parameters) proget | at Inedo.Data.SqlServerDatabaseContext.ExecuteInternal(String storedProcName, GenericDbParameter[] parameters) proget | at Inedo.Data.StrongDataReader.Read[TRow](Func`1 getReader, Boolean disposeReader)+MoveNext() proget | at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection) proget | at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source) proget | at Inedo.ProGet.Data.DB.Configuration_GetConfiguration() in C:\InedoAgent\BuildMasterTemp\192.168.44.60\Temp\_E79286\Src\ProGetCoreEx\Data\DB.cs:line 243 proget | at Inedo.ProGet.InternalConfig.ReadConfigValues() in C:\InedoAgent\BuildMasterTemp\192.168.44.60\Temp\_E79286\Src\ProGetCoreEx\Configuration\ProGetConfig.cs:line 200 proget | at Inedo.LazyCached`1.GetValue() proget | at Inedo.ProGet.ProGetConfig.Web.get_CustomErrors() in C:\InedoAgent\BuildMasterTemp\192.168.44.60\Temp\_E79286\Src\ProGetCoreEx\Configuration\ProGetConfig.Web.cs:line 63 proget | at Inedo.ProGet.WebApplication.ProGetHttpModule.Error(HttpApplication app) in C:\InedoAgent\BuildMasterTemp\192.168.44.60\Temp\_E79286\Src\ProGet.WebApplication\ProGetHttpModule.cs:line 392 proget | at Inedo.Web.InedoHttpModule.Inedo.Web.IAhWebModule.ErrorAsync(HttpApplication app) proget | at Inedo.Web.AhWebMiddleware.InvokeAsync(HttpContext context) proget | at Inedo.Web.AhWebMiddleware.InvokeAsync(HttpContext context) proget | at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application) proget | An error occurred in the web application: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached. proget | An error occurred in the web application: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached. proget | An error occurred in the web application: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached. proget | An error occurred in the web application: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached. proget | fail: Microsoft.AspNetCore.Server.Kestrel[13] proget | Connection id "0HM4OCNCDJEI2", Request id "0HM4OCNCDJEI2:00000001": An unhandled exception was thrown by the application. proget | System.InvalidOperationException: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached. proget | at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection) proget | at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions) proget | at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry) proget | at System.Data.SqlClient.SqlConnection.Open() proget | at Inedo.Data.SqlServerDatabaseContext.CreateConnection() proget | at Inedo.Data.DatabaseContext.ExecuteInternal(String storedProcName, GenericDbParameter[] parameters) proget | at Inedo.Data.SqlServerDatabaseContext.ExecuteInternal(String storedProcName, GenericDbParameter[] parameters) proget | at Inedo.Data.StrongDataReader.Read[TRow](Func`1 getReader, Boolean disposeReader)+MoveNext() proget | at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection) proget | at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source) proget | at Inedo.ProGet.Data.DB.Configuration_GetConfiguration() in C:\InedoAgent\BuildMasterTemp\192.168.44.60\Temp\_E79286\Src\ProGetCoreEx\Data\DB.cs:line 243 proget | at Inedo.ProGet.InternalConfig.ReadConfigValues() in C:\InedoAgent\BuildMasterTemp\192.168.44.60\Temp\_E79286\Src\ProGetCoreEx\Configuration\ProGetConfig.cs:line 200 proget | at Inedo.LazyCached`1.GetValue() proget | at Inedo.ProGet.ProGetConfig.Web.get_CustomErrors() in C:\InedoAgent\BuildMasterTemp\192.168.44.60\Temp\_E79286\Src\ProGetCoreEx\Configuration\ProGetConfig.Web.cs:line 63 proget | at Inedo.ProGet.WebApplication.ProGetHttpModule.Error(HttpApplication app) in C:\InedoAgent\BuildMasterTemp\192.168.44.60\Temp\_E79286\Src\ProGet.WebApplication\ProGetHttpModule.cs:line 392 proget | at Inedo.Web.InedoHttpModule.Inedo.Web.IAhWebModule.ErrorAsync(HttpApplication app) proget | at Inedo.Web.AhWebMiddleware.InvokeAsync(HttpContext context) proget | at Inedo.Web.AhWebMiddleware.InvokeAsync(HttpContext context) proget | at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)We are building our software inside docker containers, which means all NuGet packages need to be restored for each build. Sometimes we have 2-3 parallel builds. We are using proget.inedo.com/productimages/inedo/proget:5.3.16 with mcr.microsoft.com/mssql/server:2019-latest.
Any ideas how to overcome this?Best regards
Peter -
Hi Peter,
Thanks for reposting this; this error is basically what you can expect from a "network stack" being overloaded; each request to a ProGet feed can potentially open another requests to each connector configured (maybe NuGet.org, maybe others), and if you are doing a lot of requests at once, you'll get a lot of network activity queuing up. SQL Server also running on the network, so those just get added to the queue, and eventually you run out of connection.
Do you have a lot of containers running on the same computer that ProGet is running on? If so, that coudl also contribute to the network stack being overloaded.
The best way to solve is with more hardware, or by removing connectors to nuget.org, etc.
Also note that users can overload a server as well, and this is why load balancing will be very important if you want to keep reliability; See How to Prevent Server Overload in ProGet to learn more.
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login