Hi,

I upgraded today from ProGet 5.3.5 to 5.3.6 and now the webpage gives the following error:
Could not find stored procedure 'DashboardFeeds_GetWithCombinedBlobStorage'.

Here is the full stack trace if you want:

[SqlException (0x80131904): Could not find stored procedure 'DashboardFeeds_GetWithCombinedBlobStorage'.]
   System.Data.SqlClient.<>c.<ExecuteDbDataReaderAsync>b__180_0(Task`1 result) +47
   System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke() +101
   System.Threading.Tasks.Task.Execute() +70
   System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +31
   System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +60
   Inedo.Data.<CreateAsync>d__14.MoveNext() +348
   System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +31
   System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +60
   Inedo.Data.<ExecuteInternalAsync>d__37.MoveNext() +1400
   System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +31
   System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +60
   Inedo.Data.<ExecuteInternalAsync>d__7.MoveNext() +364
   System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +31
   System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +60
   Inedo.Data.<ExecuteTableAsync>d__36`1.MoveNext() +373
   System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +31
   System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +60
   Inedo.ProGet.WebApplication.Pages.<CreateBottomPanels>d__5.MoveNext() +3244
   System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +31
   System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +60
   Inedo.ProGet.WebApplication.Pages.<CreateChildControlsAsync>d__2.MoveNext() +633
   System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +31
   System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +60
   Inedo.ProGet.WebApplication.Pages.<InitializeAsync>d__3.MoveNext() +201
   System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +31
   System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +60
   Inedo.Web.PageFree.<ExecutePageLifeCycleAsync>d__46.MoveNext() +220
   System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +31
   System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +60
   Inedo.Web.PageFree.<ProcessRequestAsync>d__45.MoveNext() +323
   System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() +31
   System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +60
   System.Web.TaskAsyncHelper.EndTask(IAsyncResult ar) +59
   Inedo.Web.Handlers.AsyncHandlerWrapper.EndProcessRequest(IAsyncResult result) +33
   System.Web.CallHandlerExecutionStep.InvokeEndHandler(IAsyncResult ar) +234
   System.Web.CallHandlerExecutionStep.OnAsyncHandlerCompletion(IAsyncResult ar) +169

Is this a known issue or should I revert?

@rhessinger Hi. Unfortunately, it didn't work. Is there anything that changed in the way NuGet Restore requests are processed on the server since the last few version that could explain a timeout?

This is the exact same issue I reported a while back: https://forums.inedo.com/topic/3004/proget-timeout-at-a-specific-time

It turns out the issue is still present, but doesn't seem to happen at exactly the same time now, since we moved the build schedule. I'm guessing ProGet has a bug that does not release connections.

Related topic: https://forums.inedo.com/topic/3013

Hi,

I upgraded to 5.3.6 recently and ever since then, I've had timeout issues at around 4-5 AM eastern time.

Here is an example error:

An error occurred in the web application: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.

URL: https://localhost:444/nuget/nuget/Packages(Id='PackageName',Version='1.0.18225.1')
Referrer: (not set)
User: (unknown)
User Agent: NuGet Command Line/4.3.0 (Microsoft Windows NT 6.2.9200.0)
Stack trace: at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
at System.Data.SqlClient.SqlConnection.TryOpenInner(TaskCompletionSource`1 retry)
at System.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry)
at System.Data.SqlClient.SqlConnection.Open()
at Inedo.Data.SqlServerDatabaseContext.CreateConnection()
at Inedo.LazyAsync`1.GetValue()
at Inedo.LazyAsync`1.get_Value()
at Inedo.Data.DatabaseContext.ExecuteInternal(String storedProcName, GenericDbParameter[] parameters)
at Inedo.Data.SqlServerDatabaseContext.ExecuteInternal(String storedProcName, GenericDbParameter[] parameters)
at Inedo.Data.StrongDataReader.<Read>d__11`1.MoveNext()
at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable`1 source)
at Inedo.ProGet.Data.DB.Context.Feeds_GetFeed(Nullable`1 Feed_Id, String Feed_Name)
at Inedo.ProGet.Feeds.Feed.GetFeed(String feedName, Context externalDbContext)
at Inedo.ProGet.WebApplication.FeedEndpoints.FeedEndpointHandler.GetHttpHandler(GetHandlerParams args)
at Inedo.Web.Handlers.DynamicHttpHandling.GetHandler(HttpContext context, String requestType, String url, String pathTranslated)
at Inedo.Web.InedoHttpModule.MapHandlerAndBeginRequestAsync(HttpApplication app)
at Inedo.Web.InedoHttpModule.ProcessBegin(Object sender, EventArgs e, AsyncCallback cb, Object extraData)
at System.Web.HttpApplication.AsyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step)
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

Is this a known issue? Any idea how to fix it?

The restart seems to have done the trick. I don't see any NuGet errors since.

Thanks again!

@rhessinger I did not restart after enabling the option, I just did it this morning and will monitor. I'll get back to you with the query result if it happens again. Thanks a lot for the continued support!

I upgraded to 5.3.8 thinking the fix would work, but it seems the issue is still present, even after activating the new option and removing the Max Pool Size option.
Any idea?

This is the exact same issue I reported a while back: https://forums.inedo.com/topic/3004/proget-timeout-at-a-specific-time

It turns out the issue is still present, but doesn't seem to happen at exactly the same time now, since we moved the build schedule. I'm guessing ProGet has a bug that does not release connections.

@rhessinger So it seems like moving that scheduled trigger to 3 AM worked. There must be some kind of maintenance somewhere on the network or on the fileserver at 4AM which makes it too slow for it to work with >50 nuget restores at the same time
I will keep monitoring, but let's consider this resolved for now.

Thanks a lot!

@rhessinger There is no Scheduled job in ProGet that runs at that time. I also checked IIS and nothing seems to be set to restart at the exact time.

We did find a QA team which was triggering >50 builds at the same time at 4 AM. I will try changing the time of that script to 3 AM. If it still fails, it means that the issue is some kind of overload due to that script (which is weird because the ProGet machine is far from overloaded if we look at the CPU/RAM stats at 4 AM).

Will report back tomorrow!

Thanks again for your help and patience, very appreciated!

@rhessinger Hi. Unfortunately, it didn't work. Is there anything that changed in the way NuGet Restore requests are processed on the server since the last few version that could explain a timeout?

@rhessinger Done. Will monitor what happens tomorrow morning and report back. Thanks for the help!

@rhessinger The ProGet DB is around 15 GB and the backup seems to be at around 1 AM which is a little far from the 4 AM problem...Any other idea?

@rhessinger Hi, it resolves itself. Happends pretty much every day at around the same time. It didn't happen before the upgrade, so I don't think it's related to an external reason. We did not upgrade or change the SQL server either.

I did have an issue during the upgrade of ProGet and had to manually run inedosql for the SQL scripts.

Thanks for the help.