Posts made by carl.westman_8110

Hi Steve!

Here are all the error from yesterday

Here is the log of one of the Execution Timeout Expired messages

An error occurred in the web application: Execution Timeout Expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.

URL: http://mycompanyproget.com/
Referrer: (not set)
User: (unknown)
User Agent: AlwaysOn
IP Address: 127.0.0.1:12450
Stack trace:    at Microsoft.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at Microsoft.Data.SqlClient.TdsParserStateObject.ThrowExceptionAndWarning(Boolean callerHasConnectionLock, Boolean asyncClose)
   at Microsoft.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)
   at Microsoft.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync()
   at Microsoft.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket()
   at Microsoft.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer()
   at Microsoft.Data.SqlClient.TdsParserStateObject.TryReadByte(Byte& value)
   at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at Microsoft.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)
   at Microsoft.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean isAsync, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
   at Microsoft.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry, String method)
   at Microsoft.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry, String methodName)
   at Microsoft.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at Inedo.Data.SqlServerDatabaseContext.CreateConnection()
   at Inedo.ProGet.Data.VirtualDatabaseContext.SqlContext.CreateConnection() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E653823\Src\src\ProGet\Data\VirtualDatabaseContext.cs:line 34
   at Inedo.ProGet.Data.VirtualDatabaseContext.CreateConnection() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E653823\Src\src\ProGet\Data\VirtualDatabaseContext.cs:line 24
   at Inedo.Data.DatabaseContext.ExecuteInternal(String storedProcName, GenericDbParameter[] parameters, DatabaseCommandReturnType returnType)
   at Inedo.Data.DatabaseContext.<>c__DisplayClass34_0`1.<EnumerateTable>b__0()
   at Inedo.Data.StrongDataReader.Read[TRow](Func`1 getReader, Boolean disposeReader)+MoveNext()
   at Inedo.ProGet.FeedCache.FeedLookups..ctor(IEnumerable`1 allFeeds) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E653823\Src\src\ProGet\FeedCache.cs:line 82
   at Inedo.ProGet.FeedCache.CreateFeedLookups() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E653823\Src\src\ProGet\FeedCache.cs:line 70
   at Inedo.LazyCached`1.GetValue()
   at Inedo.LazyCached`1.get_Value()
   at Inedo.ProGet.FeedCache.GetFeeds(Boolean includeInactive) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E653823\Src\src\ProGet\FeedCache.cs:line 30
   at Inedo.ProGet.Web.Security.TaskChecker..ctor() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E653823\Src\src\ProGet\Web\Security\TaskChecker.cs:line 18
   at Inedo.ProGet.ProGetSdkConfig.Config.GetCurrentTaskChecker() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E653823\Src\src\ProGet\ProGetSdkConfig.cs:line 81
   at Inedo.Security.UserContext.<>c.<InvalidateCache>b__16_0()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
--- End of stack trace from previous location ---
   at System.Lazy`1.CreateValue()
   at Inedo.ProGet.WebApplication.Security.WebUserContext.IsAuthorizedForTask(ProGetSecuredTask task, Nullable`1 feedId) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E653823\Src\src\ProGet\WebApplication\Security\WebUserContext.cs:line 23
   at Inedo.ProGet.WebApplication.Pages.RootPage.CreateChildControlsAsync() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E653823\Src\src\ProGet\WebApplication\Pages\RootPage.cs:line 77
   at Inedo.ProGet.WebApplication.Pages.ProGetSimplePage.InitializeAsync() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E653823\Src\src\ProGet\WebApplication\Pages\ProGetSimplePage.cs:line 69
   at Inedo.Web.PageFree.SimplePageBase.ExecutePageLifeCycleAsync()
   at Inedo.Web.PageFree.SimplePageBase.ProcessRequestAsync(AhHttpContext context)
   at Inedo.Web.AhWebMiddleware.InvokeAsync(HttpContext context)

::Web Error on 06/22/2026 09:12:17::

But as you're saying, maybe its time to migrate to PostgreSQL :)

First I should apologize for any slop in this post, my own knowledge on the subject is limited so I had Opus investigate this.

But in short we've had on more than one occasion now experienced that we cannot login to Proget, when we try to access the front page (root url) we only get 500. Buuut the /health endpoint is happy go lucky and returns that everything is ok. This is a little tedious since we show the /health endpoint on our dashboards and also use that for automated restarts :p

So this is what Opus found on the issue after we couldn't connect again this morning.

Product/Version: ProGet 2025.25 (Build 11) — proget.inedo.com/productimages/inedo/proget:25.0.25, Linux container
Hosting: Azure App Service (single instance, Always On)
Database: Azure SQL Database, Standard S0 (10 DTU)

Summary

Intermittently, all web requests start failing with HTTP 500:

An error occurred in the web application: Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.

This affects both the web UI (GET /) and the Docker registry API (GET /v2/...). Once it starts, it does not self-recover; only a container restart fixes it.

Key observation — it's not the database

During the outage:

• ProGet's scheduled background tasks keep completing successfully every ~15s the entire time (Execution Dispatcher, Feed Replication, Drop Path Monitor, Node Message Cleanup, etc.). So DB connectivity clearly works.
• /health returns 200 with databaseStatus: OK, serviceStatus: OK, licenseStatus: OK.
• The Azure SQL database is idle and healthy: ~0% CPU/DTU, ≤2% workers, 0 deadlocks, 0 failed connections, ~1 session.
• The 500s return in ~3–25 ms, not after the 30s command timeout — which suggests a faulted pooled connection is being reused and rejected immediately, rather than an actual query timing out.

Only the web-request path is affected; background tasks (presumably a separate connection path/pool) are fine.

Timing / pattern

Onset is reliably around 00:00–00:05, coinciding with the burst of scheduled tasks firing at midnight. Most nights it produces a few hundred errors and recovers on its own; occasionally it wedges and stays broken for hours until restarted. Example daily 500-count (UTC): 18/6: 361, 19/6: 292, 20/6: 291, 21/6: 305, 22/6: 1177 (never recovered).

Representative log excerpt

00:01:01 Request finished GET /health - 200 648 application/json
00:01:09 An error occurred in the web application: Execution Timeout Expired. ...
00:01:09 Request finished GET / - 500 0 - 12.8ms
00:01:10 Execution Dispatcher completed. <-- background DB query OK at same moment

Opus interpretation

A transient SQL connection interruption (possibly during the midnight task burst, or an Azure SQL idle/reconfiguration disconnect) leaves a connection in the web-request pool in a faulted state. ProGet keeps reusing that pool, so every subsequent web request fails instantly with "Execution Timeout Expired," while the separately-pooled background tasks are unaffected. A restart clears the pool and restores service.

Workarounds for now

• Added ConnectRetryCount=3;ConnectRetryInterval=10 to the SQL connection string.
• Azure App Service auto-heal: recycle the app on ≥50 HTTP 500s in 5 minutes.

If required I can provide more information or logs :)

Cheers
Carl

It was hard to boil this question down to a good heading but I got the question from a colleague if there is any way to view a package and see a list of all versions regardless of feed and in there have a column which lists the different feeds the versions exists in? Something like this that Azure Artifacts does

It would be a very nice feature for anyone seeking an overview of a single package and the ability to just at a glance see which of the versions has gone on to release

Cheers
Carl

@atripp Ahh ok, maybe that was what fixed it :)

Reverting to 25.0.9 did actually work in our case So there seems to be some issue with 25.0.10

I also got an error like this

Unhandled exception: Could not find stored procedure 'Executions_FailRunningExecutions'.

@viceice does your /health also show "no problems"?

Having some wierd behaviours with Proget after updating to 25.0.10 my /health responds 200 and shows that there are no issues, so no alerts are firing

{
"applicationName": "ProGet",
"databaseStatus": "OK",
"databaseStatusDetails": null,
"extensionsInstalled": {
"ProGet": "25.0.10",
"Inedo.SDK": "3.1.0",
"Amazon AWS": "3.1.3",
"InedoCore": "3.0.5",
"Azure": "3.1.2"
},
"licenseStatus": "OK",
"licenseStatusDetail": null,
"versionNumber": "2025.10 (Build 11)",
"releaseNumber": "25.0.10",
"startTime": 1758271289653,
"serviceStatus": "OK",
"serviceStatusDetail": null,
"replicationStatus": {
"serverStatus": null,
"serverError": null,
"clientStatus": null,
"clienterror": null,
"incoming": [],
"outgoing": []
}
}

But trying to go to the website I only get http error 500

The logs show an error "Could not find stored procedure 'Security_GetRoles'

2025-09-19T09:04:08.8094686Z info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2025-09-19T09:04:08.8095216Z       Request starting HTTP/1.1 GET http://proget.dev.companyname.com/health - - -
2025-09-19T09:04:08.8097717Z info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2025-09-19T09:04:08.8099365Z       Request finished HTTP/1.1 GET http://proget.dev.companyname.com/health - 200 619 application/json 0.3870ms
2025-09-19T09:04:10.6672063Z info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2025-09-19T09:04:10.6672953Z       Request starting HTTP/1.1 GET http://proget.dev.companyname.com/ - - -
2025-09-19T09:04:10.6687087Z An error occurred in the web application: Could not find stored procedure 'Security_GetRoles'.
2025-09-19T09:04:10.6778579Z info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2025-09-19T09:04:10.6779411Z       Request finished HTTP/1.1 GET http://proget.dev.companyname.com/ - 500 0 - 10.6553ms
2025-09-19T09:04:13.8057598Z info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2025-09-19T09:04:13.8058173Z       Request starting HTTP/1.1 GET http://proget.dev.companyname.com/health - - -
2025-09-19T09:04:13.8064277Z info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2025-09-19T09:04:13.8064479Z       Request finished HTTP/1.1 GET http://proget.dev.companyname.com/health - 200 619 application/json 0.4290ms
2025-09-19T09:04:13.9852857Z info: Microsoft.AspNetCore.Hosting.Diagnostics[1]

Anyone stumbled upon something similar and have any ideas on how to fix this?

Cheers
Carl

@apxltd I understand what you mean

When I have your attention I would like to just tell you that you have done an amazing work on ProGet as a product. It has everything we need while not being overcomplicated or overpriced. It really is a great product.

And the support through this forum is also always superb, fast and accurate.

Great work!

@apxltd I see, thanks for the info!

The use cases seems legit I think, like bundling multiple packages. Why would you like to have it undone?

I stumbled upon this V symbol when scrolling through Inedos open feeds. What does it mean and how does one add it? :)

@dean-houston

Ahh, thanks for pointing that out :)

As an answer to my question about health (and for anyone coming here in the future), I should've just read the docs properly, it was very clear what to check I love Proget! These things just exists and are well documented 🥰

https://docs.inedo.com/docs/proget-reference-api-health

Hi there!

We run Proget as an Azure Container Web App atm with great success. To this we have an Azure SQL DB and we have mounted an Azure storage account file share at /var/proget to get all the files stored in Azure and backed up there. Everything deployed with Bicep, which is awsome

There are some cool features for Azure Web Apps though that I would like to utilize to maximize the robustness and up time of our Proget instance for our developers.

One such feature is the "scale out" together with "Health check". This (if I understand correctly) would allow me to run multiple instances of the Proget container and Azure will ping them to see if they are healthy. If one is deemed unhealthy it will be brought down and replaced.

I however have some concerns about what happens if I scale out the web app to multiple instances but they all have the same SQL database and file share? The file share shouldn't be an issue, but what about the database? Can it handle writing and reading from multiple instances of an Azure Container Web App? Do you have any experience or best practices regarding this? I'm not very experienced with DB:s, and my biggest concern is that there will be any deadlock, corrupt data and overall breakage of the data in the DB, but maybe that is not a concern.

Also when on the topic, do you have any recommended path to probe/ping for checking health of both the Proget instance and the SQL DB connection?

I guess some endpoint which tries to read something from the DB and returns a non 200-299 code if the read fails would be good. It should be something "static" and not a path to a package which could be removed any time in the future.

(If there are concerns with the DB I guess a single instance with "auto replace" by the health check is good enough, but this would cause a one hour downtime which would be nice to not have )

It should be noted that we do not have any kind of load on Proget since we are pretty few devs, I would only like to use these features to increase the uptime and robustness of the app since people will come and complain at me if Proget is down

Cheers
Carl

@atripp, I can confirm that 2024.1 works great

Awsome @atripp, thanks for the quick support!

@atripp

Yep, it's there

schema_id = 4
is_ms_shipped = True

I'm no database master of any kind, but I think I've chosen all defaults when creating both the Az sql server and the db

Hi @atripp,

When creating a new empty Azure SQL DB this seems to be there by default.

It's also present in my Proget DB

Hi there!

Trying to update my Proget instance which is running as an Azure Container Web App from 23.0.18 to 24.0.0 but I get some weird error when the container is trying to start, see below.

2024-04-23T12:32:55.267041815Z Unhandled exception: Microsoft.Data.SqlClient.SqlException (0x80131904): Cannot drop the view 'database_firewall_rules', because it does not exist or you do not have permission.
2024-04-23T12:32:55.267114815Z    at Microsoft.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
2024-04-23T12:32:55.267119415Z    at Microsoft.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
2024-04-23T12:32:55.267122715Z    at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, SqlCommand command, Boolean callerHasConnectionLock, Boolean asyncClose)
2024-04-23T12:32:55.267126315Z    at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
2024-04-23T12:32:55.267133315Z    at Microsoft.Data.SqlClient.SqlCommand.RunExecuteNonQueryTds(String methodName, Boolean isAsync, Int32 timeout, Boolean asyncWrite)
2024-04-23T12:32:55.267141515Z    at Microsoft.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry, String methodName)
2024-04-23T12:32:55.267145115Z    at Microsoft.Data.SqlClient.SqlCommand.ExecuteNonQuery()
2024-04-23T12:32:55.267148215Z    at Inedo.DbUpdater.SqlServer.SqlServerDatabaseConnection.ExecuteQueryWithSplitter(String query, SqlTransaction transaction)
2024-04-23T12:32:55.267151315Z    at Inedo.DbUpdater.SqlServer.SqlServerDatabaseConnection.ExecuteScripts(IEnumerable`1 scripts, ChangeScriptState state)
2024-04-23T12:32:55.267154415Z    at Inedo.DbUpdater.ConsoleHost.Update(String scriptPath, String connectionString, Boolean force)
2024-04-23T12:32:55.267157515Z    at Inedo.DbUpdater.ConsoleHost.Run(ArgList args)
2024-04-23T12:32:55.267160515Z    at Inedo.DbUpdater.ConsoleHost.RunAsync(IEnumerable`1 args)
2024-04-23T12:32:55.267163615Z    at Inedo.ProGet.Service.Program.UpdateDatabaseSchemaAsync() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E438586\Src\ProGet.Service\Program.cs:line 318
2024-04-23T12:32:55.267169215Z    at Inedo.ProGet.Service.Program.Run(WebServerMode mode, Boolean runAsService, Boolean linuxContainer, String urls) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E438586\Src\ProGet.Service\Program.cs:line 28
2024-04-23T12:32:55.267173015Z    at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
2024-04-23T12:32:55.267176215Z    at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs, BindingFlags invokeAttr)

I'm using an Azure sql database which have worked fine with 23.0.18

Cheers
Carl

I understand :) thanks for the quick answer!

Just want to say that Proget really is a super product :) been stuck with Azure Artifacts for a while and getting into a self hosted Proget instance really is a relief, all the features I was missing in ADO Artifacts now work as expected and we can actually use Universal Packages :) So thanks for that, keep up the good work!

Just wanted to post a feature request

If in a universal package in proget and you press Files and then Preview on a folder and that folder contains an index.html it would be frikkin awsome if Proget could open a new tab with the whole documentation vizualized

This would mean that we could use proget both for storing docs artifacts as well as for people to watch the documentation

As it is today we have to publish our sphinx or mkdocs htmls both as an artifact to Proget with the correct version as well as to a webserver which hosts the documentation where people can watch it.

Cheers
Carl

Hi Dan!

My IT department solved the issue for now by ignoring certificate errors for LDAPS as per your suggestion above while it would be nice to get it to work "correctly" i'll wait until there is a guide available on how to set it up to not have to ignore the errors Many thanks for all your help!

For future reference and people

Ahh sorry, should have mentioned that, I'm running Proget as an Azure App Service yes :)

Thanks @Dan_Woolf!

Will try to add the certificate according to the guide :) do I need to follow this guide after I have the certificate available in the web app env?

https://docs.inedo.com/docs/installation-linux-https-support#configuring-https-without-a-reverse-proxy

Hi there!

We're trying to setup LDAPS configuration towards Azure AD so that we can put the right permission on people when they login with SAML to Proget :) but we get some wierd error messages right now which we cannot figure out, anyone here have any idea? See the picture below.

Fyi, we are hosting our Proget as a web app in Azure with the Docker container and a functioning certificate.

Kind regards
Carl

Ahh that could be it, I will update to latest version and try it again :)

@stevedennis said in Azure blob storage gives 500 internal server error:

Admin

Hi Steve! Here is the full message, can you see anything specific in it? Or is it to general?

System.NullReferenceException: Object reference not set to an instance of an object.
at Azure.Storage.Blobs.Models.BlobOpenWriteOptionsExtensions.ToBlockBlobOpenWriteOptions(BlobOpenWriteOptions options)
at Azure.Storage.Blobs.BlobClient.OpenWriteInternal(Boolean overwrite, BlobOpenWriteOptions options, Boolean async, CancellationToken cancellationToken)
at Azure.Storage.Blobs.BlobClient.OpenWriteAsync(Boolean overwrite, BlobOpenWriteOptions options, CancellationToken cancellationToken)
at Inedo.ProGet.Extensions.FileSystems.CaseInsensitiveFileSystem.CreateFileAsync(String fileName, FileAccessHints hints, CancellationToken cancellationToken) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E233598\Src\ProGetCoreEx\Extensions\FileSystems\CaseInsensitiveFileSystem.cs:line 66
at Inedo.ProGet.Feeds.Universal.UniversalFeed.InstallPackageInternalAsync(Stream packageStream, InstallPackageOptions options, Boolean promoted) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E233598\Src\ProGetCoreEx\Feeds\Universal\UniversalFeed.cs:line 743
at Inedo.ProGet.WebApplication.FeedEndpoints.Universal.UniversalRepositoryHandler.ProcessUploadRequestAsync(UniversalFeed feed, HttpContext context, WebApiContext apiContext, String url) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E233598\Src\ProGet.WebApplication\FeedEndpoints\Universal\UniversalRepositoryHandler.Upload.cs:line 76
at Inedo.ProGet.WebApplication.FeedEndpoints.FeedEndpointHandler.FeedRequestHandler.ProcessRequestAsync(HttpContext context) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E233598\Src\ProGet.WebApplication\FeedEndpoints\FeedEndpointHandler.cs:line 123

I've configured a Universal feed with Azure blob storage according to the guide here
https://docs.inedo.com/docs/proget-howto-configure-feed-for-azure-blob

But when trying to upload a package I only get the following

If I change back to default storage the error disappears.

Anyone got any ideas?