Hi there!

We run Proget as an Azure Container Web App atm with great success. To this we have an Azure SQL DB and we have mounted an Azure storage account file share at /var/proget to get all the files stored in Azure and backed up there. Everything deployed with Bicep, which is awsome

There are some cool features for Azure Web Apps though that I would like to utilize to maximize the robustness and up time of our Proget instance for our developers.

One such feature is the "scale out" together with "Health check". This (if I understand correctly) would allow me to run multiple instances of the Proget container and Azure will ping them to see if they are healthy. If one is deemed unhealthy it will be brought down and replaced.

I however have some concerns about what happens if I scale out the web app to multiple instances but they all have the same SQL database and file share? The file share shouldn't be an issue, but what about the database? Can it handle writing and reading from multiple instances of an Azure Container Web App? Do you have any experience or best practices regarding this? I'm not very experienced with DB:s, and my biggest concern is that there will be any deadlock, corrupt data and overall breakage of the data in the DB, but maybe that is not a concern.

Also when on the topic, do you have any recommended path to probe/ping for checking health of both the Proget instance and the SQL DB connection?

I guess some endpoint which tries to read something from the DB and returns a non 200-299 code if the read fails would be good. It should be something "static" and not a path to a package which could be removed any time in the future.

(If there are concerns with the DB I guess a single instance with "auto replace" by the health check is good enough, but this would cause a one hour downtime which would be nice to not have )

It should be noted that we do not have any kind of load on Proget since we are pretty few devs, I would only like to use these features to increase the uptime and robustness of the app since people will come and complain at me if Proget is down

Cheers
Carl

Hi Steve!

Here are all the error from yesterday

Here is the log of one of the Execution Timeout Expired messages

An error occurred in the web application: Execution Timeout Expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.

URL: http://mycompanyproget.com/
Referrer: (not set)
User: (unknown)
User Agent: AlwaysOn
IP Address: 127.0.0.1:12450
Stack trace:    at Microsoft.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at Microsoft.Data.SqlClient.TdsParserStateObject.ThrowExceptionAndWarning(Boolean callerHasConnectionLock, Boolean asyncClose)
   at Microsoft.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)
   at Microsoft.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync()
   at Microsoft.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket()
   at Microsoft.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer()
   at Microsoft.Data.SqlClient.TdsParserStateObject.TryReadByte(Byte& value)
   at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at Microsoft.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)
   at Microsoft.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean isAsync, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
   at Microsoft.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry, String method)
   at Microsoft.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry, String methodName)
   at Microsoft.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at Inedo.Data.SqlServerDatabaseContext.CreateConnection()
   at Inedo.ProGet.Data.VirtualDatabaseContext.SqlContext.CreateConnection() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E653823\Src\src\ProGet\Data\VirtualDatabaseContext.cs:line 34
   at Inedo.ProGet.Data.VirtualDatabaseContext.CreateConnection() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E653823\Src\src\ProGet\Data\VirtualDatabaseContext.cs:line 24
   at Inedo.Data.DatabaseContext.ExecuteInternal(String storedProcName, GenericDbParameter[] parameters, DatabaseCommandReturnType returnType)
   at Inedo.Data.DatabaseContext.<>c__DisplayClass34_0`1.<EnumerateTable>b__0()
   at Inedo.Data.StrongDataReader.Read[TRow](Func`1 getReader, Boolean disposeReader)+MoveNext()
   at Inedo.ProGet.FeedCache.FeedLookups..ctor(IEnumerable`1 allFeeds) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E653823\Src\src\ProGet\FeedCache.cs:line 82
   at Inedo.ProGet.FeedCache.CreateFeedLookups() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E653823\Src\src\ProGet\FeedCache.cs:line 70
   at Inedo.LazyCached`1.GetValue()
   at Inedo.LazyCached`1.get_Value()
   at Inedo.ProGet.FeedCache.GetFeeds(Boolean includeInactive) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E653823\Src\src\ProGet\FeedCache.cs:line 30
   at Inedo.ProGet.Web.Security.TaskChecker..ctor() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E653823\Src\src\ProGet\Web\Security\TaskChecker.cs:line 18
   at Inedo.ProGet.ProGetSdkConfig.Config.GetCurrentTaskChecker() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E653823\Src\src\ProGet\ProGetSdkConfig.cs:line 81
   at Inedo.Security.UserContext.<>c.<InvalidateCache>b__16_0()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
--- End of stack trace from previous location ---
   at System.Lazy`1.CreateValue()
   at Inedo.ProGet.WebApplication.Security.WebUserContext.IsAuthorizedForTask(ProGetSecuredTask task, Nullable`1 feedId) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E653823\Src\src\ProGet\WebApplication\Security\WebUserContext.cs:line 23
   at Inedo.ProGet.WebApplication.Pages.RootPage.CreateChildControlsAsync() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E653823\Src\src\ProGet\WebApplication\Pages\RootPage.cs:line 77
   at Inedo.ProGet.WebApplication.Pages.ProGetSimplePage.InitializeAsync() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E653823\Src\src\ProGet\WebApplication\Pages\ProGetSimplePage.cs:line 69
   at Inedo.Web.PageFree.SimplePageBase.ExecutePageLifeCycleAsync()
   at Inedo.Web.PageFree.SimplePageBase.ProcessRequestAsync(AhHttpContext context)
   at Inedo.Web.AhWebMiddleware.InvokeAsync(HttpContext context)

::Web Error on 06/22/2026 09:12:17::

But as you're saying, maybe its time to migrate to PostgreSQL :)

First I should apologize for any slop in this post, my own knowledge on the subject is limited so I had Opus investigate this.

But in short we've had on more than one occasion now experienced that we cannot login to Proget, when we try to access the front page (root url) we only get 500. Buuut the /health endpoint is happy go lucky and returns that everything is ok. This is a little tedious since we show the /health endpoint on our dashboards and also use that for automated restarts :p

So this is what Opus found on the issue after we couldn't connect again this morning.

Product/Version: ProGet 2025.25 (Build 11) — proget.inedo.com/productimages/inedo/proget:25.0.25, Linux container
Hosting: Azure App Service (single instance, Always On)
Database: Azure SQL Database, Standard S0 (10 DTU)

Summary

Intermittently, all web requests start failing with HTTP 500:

An error occurred in the web application: Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.

This affects both the web UI (GET /) and the Docker registry API (GET /v2/...). Once it starts, it does not self-recover; only a container restart fixes it.

Key observation — it's not the database

During the outage:

• ProGet's scheduled background tasks keep completing successfully every ~15s the entire time (Execution Dispatcher, Feed Replication, Drop Path Monitor, Node Message Cleanup, etc.). So DB connectivity clearly works.
• /health returns 200 with databaseStatus: OK, serviceStatus: OK, licenseStatus: OK.
• The Azure SQL database is idle and healthy: ~0% CPU/DTU, ≤2% workers, 0 deadlocks, 0 failed connections, ~1 session.
• The 500s return in ~3–25 ms, not after the 30s command timeout — which suggests a faulted pooled connection is being reused and rejected immediately, rather than an actual query timing out.

Only the web-request path is affected; background tasks (presumably a separate connection path/pool) are fine.

Timing / pattern

Onset is reliably around 00:00–00:05, coinciding with the burst of scheduled tasks firing at midnight. Most nights it produces a few hundred errors and recovers on its own; occasionally it wedges and stays broken for hours until restarted. Example daily 500-count (UTC): 18/6: 361, 19/6: 292, 20/6: 291, 21/6: 305, 22/6: 1177 (never recovered).

Representative log excerpt

00:01:01 Request finished GET /health - 200 648 application/json
00:01:09 An error occurred in the web application: Execution Timeout Expired. ...
00:01:09 Request finished GET / - 500 0 - 12.8ms
00:01:10 Execution Dispatcher completed. <-- background DB query OK at same moment

Opus interpretation

A transient SQL connection interruption (possibly during the midnight task burst, or an Azure SQL idle/reconfiguration disconnect) leaves a connection in the web-request pool in a faulted state. ProGet keeps reusing that pool, so every subsequent web request fails instantly with "Execution Timeout Expired," while the separately-pooled background tasks are unaffected. A restart clears the pool and restores service.

Workarounds for now

• Added ConnectRetryCount=3;ConnectRetryInterval=10 to the SQL connection string.
• Azure App Service auto-heal: recycle the app on ≥50 HTTP 500s in 5 minutes.

If required I can provide more information or logs :)

Cheers
Carl

It was hard to boil this question down to a good heading but I got the question from a colleague if there is any way to view a package and see a list of all versions regardless of feed and in there have a column which lists the different feeds the versions exists in? Something like this that Azure Artifacts does

It would be a very nice feature for anyone seeking an overview of a single package and the ability to just at a glance see which of the versions has gone on to release

Cheers
Carl

@atripp Ahh ok, maybe that was what fixed it :)

Reverting to 25.0.9 did actually work in our case So there seems to be some issue with 25.0.10

I also got an error like this

Unhandled exception: Could not find stored procedure 'Executions_FailRunningExecutions'.

@viceice does your /health also show "no problems"?

Having some wierd behaviours with Proget after updating to 25.0.10 my /health responds 200 and shows that there are no issues, so no alerts are firing

{
"applicationName": "ProGet",
"databaseStatus": "OK",
"databaseStatusDetails": null,
"extensionsInstalled": {
"ProGet": "25.0.10",
"Inedo.SDK": "3.1.0",
"Amazon AWS": "3.1.3",
"InedoCore": "3.0.5",
"Azure": "3.1.2"
},
"licenseStatus": "OK",
"licenseStatusDetail": null,
"versionNumber": "2025.10 (Build 11)",
"releaseNumber": "25.0.10",
"startTime": 1758271289653,
"serviceStatus": "OK",
"serviceStatusDetail": null,
"replicationStatus": {
"serverStatus": null,
"serverError": null,
"clientStatus": null,
"clienterror": null,
"incoming": [],
"outgoing": []
}
}

But trying to go to the website I only get http error 500

The logs show an error "Could not find stored procedure 'Security_GetRoles'

2025-09-19T09:04:08.8094686Z info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2025-09-19T09:04:08.8095216Z       Request starting HTTP/1.1 GET http://proget.dev.companyname.com/health - - -
2025-09-19T09:04:08.8097717Z info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2025-09-19T09:04:08.8099365Z       Request finished HTTP/1.1 GET http://proget.dev.companyname.com/health - 200 619 application/json 0.3870ms
2025-09-19T09:04:10.6672063Z info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2025-09-19T09:04:10.6672953Z       Request starting HTTP/1.1 GET http://proget.dev.companyname.com/ - - -
2025-09-19T09:04:10.6687087Z An error occurred in the web application: Could not find stored procedure 'Security_GetRoles'.
2025-09-19T09:04:10.6778579Z info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2025-09-19T09:04:10.6779411Z       Request finished HTTP/1.1 GET http://proget.dev.companyname.com/ - 500 0 - 10.6553ms
2025-09-19T09:04:13.8057598Z info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2025-09-19T09:04:13.8058173Z       Request starting HTTP/1.1 GET http://proget.dev.companyname.com/health - - -
2025-09-19T09:04:13.8064277Z info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2025-09-19T09:04:13.8064479Z       Request finished HTTP/1.1 GET http://proget.dev.companyname.com/health - 200 619 application/json 0.4290ms
2025-09-19T09:04:13.9852857Z info: Microsoft.AspNetCore.Hosting.Diagnostics[1]

Anyone stumbled upon something similar and have any ideas on how to fix this?

Cheers
Carl

@apxltd I understand what you mean

When I have your attention I would like to just tell you that you have done an amazing work on ProGet as a product. It has everything we need while not being overcomplicated or overpriced. It really is a great product.

And the support through this forum is also always superb, fast and accurate.

Great work!

@apxltd I see, thanks for the info!

The use cases seems legit I think, like bundling multiple packages. Why would you like to have it undone?