ProGet timeout at a specific time

gravufo

Related topic: https://forums.inedo.com/topic/3013

Hi,

I upgraded to 5.3.6 recently and ever since then, I've had timeout issues at around 4-5 AM eastern time.

Here is an example error:

An error occurred in the web application: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.

URL: https://localhost:444/nuget/nuget/Packages(Id='PackageName',Version='1.0.18225.1')
Referrer: (not set)
User: (unknown)
User Agent: NuGet Command Line/4.3.0 (Microsoft Windows NT 6.2.9200.0)
Stack trace: at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
at System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource`1 retry)
at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)
at System.Data.SqlClient.SqlConnection.Open()
at Inedo.Data.SqlServerDatabaseContext.CreateConnection()
at Inedo.LazyAsync`1.GetValue()
at Inedo.LazyAsync`1.get_Value()
at Inedo.Data.DatabaseContext.ExecuteInternal(String storedProcName, GenericDbParameter[] parameters)
at Inedo.Data.SqlServerDatabaseContext.ExecuteInternal(String storedProcName, GenericDbParameter[] parameters)
at Inedo.Data.StrongDataReader.<Read>d__11`1.MoveNext()
at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable`1 source)
at Inedo.ProGet.Data.DB.Context.Feeds_GetFeed(Nullable`1 Feed_Id, String Feed_Name)
at Inedo.ProGet.Feeds.Feed.GetFeed(String feedName, Context externalDbContext)
at Inedo.ProGet.WebApplication.FeedEndpoints.FeedEndpointHandler.GetHttpHandler(GetHandlerParams args)
at Inedo.Web.Handlers.DynamicHttpHandling.GetHandler(HttpContext context, String requestType, String url, String pathTranslated)
at Inedo.Web.InedoHttpModule.MapHandlerAndBeginRequestAsync(HttpApplication app)
at Inedo.Web.InedoHttpModule.ProcessBegin(Object sender, EventArgs e, AsyncCallback cb, Object extraData)
at System.Web.HttpApplication.AsyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step)
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

Is this a known issue? Any idea how to fix it?

rhessinger

Hi @gravufo,

When this happens, does it resolve itself or do you have to restart ProGet for it to start working again?

Thanks,
Rich

gravufo

@rhessinger Hi, it resolves itself. Happends pretty much every day at around the same time. It didn't happen before the upgrade, so I don't think it's related to an external reason. We did not upgrade or change the SQL server either.

I did have an issue during the upgrade of ProGet and had to manually run inedosql for the SQL scripts.

Thanks for the help.

rhessinger

Hi @gravufo,

I do not think the install issue would have caused any issues (especially because it resolves itself). Do you have any server backup processes or SQL database backups going on at that time? I have seen issues where certain backup software will lock up SQL server during backups (especially SQL servers with large databases) .

Thanks,
Rich

gravufo

@rhessinger The ProGet DB is around 15 GB and the backup seems to be at around 1 AM which is a little far from the 4 AM problem...Any other idea?

rhessinger

Hi @gravufo,

Could you try increasing the Max Pool Size to 300 in your SQL connection string? I would like to see if this is an issue with the number of SQL connections.

Thanks,
Rich

gravufo

@rhessinger Done. Will monitor what happens tomorrow morning and report back. Thanks for the help!

rhessinger

Hi @gravufo,

No problem, please let me know either way. Thanks!

Thanks,
Rich

gravufo

@rhessinger Hi. Unfortunately, it didn't work. Is there anything that changed in the way NuGet Restore requests are processed on the server since the last few version that could explain a timeout?

rhessinger

Hi @gravufo,

There is nothing that changed in the NuGet API that would cause SQL connection issues. We did add the ability to enable the NuGet v3 API in addition to the V2 API, but you would have to enable V3 manually for your existing feeds. Still, that does not affect open a connection to SQL. What is weird to me is that this happens every day at the same time. This screams like there is something scheduled. Can you check your Application Pool in IIS and verify you don't have an application pool restart happening at that time?

Thanks,
Rich

rhessinger

Hi @gravufo,

Can you please check your scheduled jobs in the Administration page of ProGet. It could be possible all of the jobs are running at 4:00 am.

Thanks,
Rich

gravufo

@rhessinger There is no Scheduled job in ProGet that runs at that time. I also checked IIS and nothing seems to be set to restart at the exact time.

We did find a QA team which was triggering >50 builds at the same time at 4 AM. I will try changing the time of that script to 3 AM. If it still fails, it means that the issue is some kind of overload due to that script (which is weird because the ProGet machine is far from overloaded if we look at the CPU/RAM stats at 4 AM).

Will report back tomorrow!

Thanks again for your help and patience, very appreciated!

rhessinger

Hi @gravufo,

No problem. It does seem weird that >50 builds would cause that. Unless if it is the SQL server that is getting overwhelmed at that time.

Thanks,
Rich

gravufo

@rhessinger So it seems like moving that scheduled trigger to 3 AM worked. There must be some kind of maintenance somewhere on the network or on the fileserver at 4AM which makes it too slow for it to work with >50 nuget restores at the same time
I will keep monitoring, but let's consider this resolved for now.

Thanks a lot!

rhessinger

Hi @gravufo,

Awesome! Just reply back here if it does start up again.

Thanks,
Rich