Thanks @rhessinger for the quick response and quick fix! One of the many reasons we choose Proget

Thanks @rhessinger

We've imported the packages in the meantime, so we're not blocked.

For reference, the Azure side answers our requests fine, including with Accept: application/vnd.pypi.simple.v1+json, where the JSON contains the files list with url and hashes.sha256 for every file.

Cheers
Carl

Hi there! I have some problems hooking up our Azure Artifacts pypi feed to Proget for a smooth transition from Azure Artifacts to Proget, wonder if you have some input :)

ProGet 2026.11 (Build 19), Basic Edition. PyPI feed with a connector to an Azure Artifacts PyPI feed:

• Connector URL: https://pkgs.dev.azure.com/<org>/_packaging/<feed>/pypi (simple path left at the default simple/)
• Authentication: Basic, user name azure, password = Azure DevOps PAT with Packaging (Read)

Requesting a package through the ProGet feed returns "Package not found", and the Diagnostic Center logs:

Connector error: The remote server returned an error: (401) Unauthorized.
System.Net.WebException ... at Inedo.ProGet.Feeds.Pypi.PypiConnector.GetSimpleProjectDetailAsync(String projectName, ...) PypiConnector.cs:line 102

The same request from the ProGet host's network with curl succeeds: curl -u azure:<PAT> https://pkgs.dev.azure.com/<org>/_packaging/<feed>/pypi/simple/<package>/ returns 200, for both the HTML and the application/vnd.pypi.simple.v1+json Accept header. With a valid PAT Azure returns 404 for wrong paths, so the 401 means the connector's credentials are not accepted. Re-entering the password in the UI and via the management API makes no difference. Azure's 401 challenge lists three schemes in this order: WWW-Authenticate: Bearer authorization_uri=..., Basic realm="...", TFS-Federated. Is it possible the Basic credentials are only sent after a challenge and the Bearer entry is picked instead? Is there a way to make the PyPI connector send Basic credentials preemptively?

Cheers
Carl

@atripp award for fastest forum response goes to you yep that seems to have solved it :) simple enough ;) Thanks!

Maybe it would be an idea to highlight or add a little orange "stalled" badge on the ones in the list that causes the warning? Just a nice to have.

Cheers
Carl

This message popped up today, clicking it takes me to the Service Status page, which gives me nothing :p do you have any idea how to proceed and find the culprit? :)

@stevedennis sounds perfect, thanks Steve! :)

We're running ProGet 25.0.25 in Docker (Azure App Service for Containers,
configured via environment variables, so no shared config file).

This instance has never had an encryption key set. The startup log confirms
it: "No encryption key is configured. Credentials will be stored in plain
text." We have a couple of feed connectors whose credentials are therefore
stored unencrypted, and we'd like to fix that.

The documentation I can find only covers preserving an existing key across a
migration (Backing Up & Restoring, Migrate to a New Server), and the
configuration files reference just notes that EncryptionKey is optional. I
can't find anything on introducing a key to an instance that doesn't have
one.

My questions:

1. Is it supported to add an EncryptionKey to an existing instance that has
   never had one?

2. What happens to credentials already stored in plaintext? Are they
   migrated/re-encrypted on startup, left as-is, or will reads fail once a
   key is configured?

3. If there's no automatic migration, is re-entering each credential through
   the UI after setting the key the correct approach?

4. We're also migrating this instance from SQL Server to PostgreSQL 17
   shortly. Is there a preferred order - set the key before or after the
   database migration, or does it not matter?

Thanks!

Hi Steve!

Here are all the error from yesterday

Here is the log of one of the Execution Timeout Expired messages

An error occurred in the web application: Execution Timeout Expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.

URL: http://mycompanyproget.com/
Referrer: (not set)
User: (unknown)
User Agent: AlwaysOn
IP Address: 127.0.0.1:12450
Stack trace:    at Microsoft.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at Microsoft.Data.SqlClient.TdsParserStateObject.ThrowExceptionAndWarning(Boolean callerHasConnectionLock, Boolean asyncClose)
   at Microsoft.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)
   at Microsoft.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync()
   at Microsoft.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket()
   at Microsoft.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer()
   at Microsoft.Data.SqlClient.TdsParserStateObject.TryReadByte(Byte& value)
   at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at Microsoft.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)
   at Microsoft.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean isAsync, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
   at Microsoft.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry, String method)
   at Microsoft.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry, String methodName)
   at Microsoft.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at Inedo.Data.SqlServerDatabaseContext.CreateConnection()
   at Inedo.ProGet.Data.VirtualDatabaseContext.SqlContext.CreateConnection() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E653823\Src\src\ProGet\Data\VirtualDatabaseContext.cs:line 34
   at Inedo.ProGet.Data.VirtualDatabaseContext.CreateConnection() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E653823\Src\src\ProGet\Data\VirtualDatabaseContext.cs:line 24
   at Inedo.Data.DatabaseContext.ExecuteInternal(String storedProcName, GenericDbParameter[] parameters, DatabaseCommandReturnType returnType)
   at Inedo.Data.DatabaseContext.<>c__DisplayClass34_0`1.<EnumerateTable>b__0()
   at Inedo.Data.StrongDataReader.Read[TRow](Func`1 getReader, Boolean disposeReader)+MoveNext()
   at Inedo.ProGet.FeedCache.FeedLookups..ctor(IEnumerable`1 allFeeds) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E653823\Src\src\ProGet\FeedCache.cs:line 82
   at Inedo.ProGet.FeedCache.CreateFeedLookups() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E653823\Src\src\ProGet\FeedCache.cs:line 70
   at Inedo.LazyCached`1.GetValue()
   at Inedo.LazyCached`1.get_Value()
   at Inedo.ProGet.FeedCache.GetFeeds(Boolean includeInactive) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E653823\Src\src\ProGet\FeedCache.cs:line 30
   at Inedo.ProGet.Web.Security.TaskChecker..ctor() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E653823\Src\src\ProGet\Web\Security\TaskChecker.cs:line 18
   at Inedo.ProGet.ProGetSdkConfig.Config.GetCurrentTaskChecker() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E653823\Src\src\ProGet\ProGetSdkConfig.cs:line 81
   at Inedo.Security.UserContext.<>c.<InvalidateCache>b__16_0()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
--- End of stack trace from previous location ---
   at System.Lazy`1.CreateValue()
   at Inedo.ProGet.WebApplication.Security.WebUserContext.IsAuthorizedForTask(ProGetSecuredTask task, Nullable`1 feedId) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E653823\Src\src\ProGet\WebApplication\Security\WebUserContext.cs:line 23
   at Inedo.ProGet.WebApplication.Pages.RootPage.CreateChildControlsAsync() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E653823\Src\src\ProGet\WebApplication\Pages\RootPage.cs:line 77
   at Inedo.ProGet.WebApplication.Pages.ProGetSimplePage.InitializeAsync() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E653823\Src\src\ProGet\WebApplication\Pages\ProGetSimplePage.cs:line 69
   at Inedo.Web.PageFree.SimplePageBase.ExecutePageLifeCycleAsync()
   at Inedo.Web.PageFree.SimplePageBase.ProcessRequestAsync(AhHttpContext context)
   at Inedo.Web.AhWebMiddleware.InvokeAsync(HttpContext context)

::Web Error on 06/22/2026 09:12:17::

But as you're saying, maybe its time to migrate to PostgreSQL :)

First I should apologize for any slop in this post, my own knowledge on the subject is limited so I had Opus investigate this.

But in short we've had on more than one occasion now experienced that we cannot login to Proget, when we try to access the front page (root url) we only get 500. Buuut the /health endpoint is happy go lucky and returns that everything is ok. This is a little tedious since we show the /health endpoint on our dashboards and also use that for automated restarts :p

So this is what Opus found on the issue after we couldn't connect again this morning.

Product/Version: ProGet 2025.25 (Build 11) — proget.inedo.com/productimages/inedo/proget:25.0.25, Linux container
Hosting: Azure App Service (single instance, Always On)
Database: Azure SQL Database, Standard S0 (10 DTU)

Summary

Intermittently, all web requests start failing with HTTP 500:

An error occurred in the web application: Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.

This affects both the web UI (GET /) and the Docker registry API (GET /v2/...). Once it starts, it does not self-recover; only a container restart fixes it.

Key observation — it's not the database

During the outage:

• ProGet's scheduled background tasks keep completing successfully every ~15s the entire time (Execution Dispatcher, Feed Replication, Drop Path Monitor, Node Message Cleanup, etc.). So DB connectivity clearly works.
• /health returns 200 with databaseStatus: OK, serviceStatus: OK, licenseStatus: OK.
• The Azure SQL database is idle and healthy: ~0% CPU/DTU, ≤2% workers, 0 deadlocks, 0 failed connections, ~1 session.
• The 500s return in ~3–25 ms, not after the 30s command timeout — which suggests a faulted pooled connection is being reused and rejected immediately, rather than an actual query timing out.

Only the web-request path is affected; background tasks (presumably a separate connection path/pool) are fine.

Timing / pattern

Onset is reliably around 00:00–00:05, coinciding with the burst of scheduled tasks firing at midnight. Most nights it produces a few hundred errors and recovers on its own; occasionally it wedges and stays broken for hours until restarted. Example daily 500-count (UTC): 18/6: 361, 19/6: 292, 20/6: 291, 21/6: 305, 22/6: 1177 (never recovered).

Representative log excerpt

00:01:01 Request finished GET /health - 200 648 application/json
00:01:09 An error occurred in the web application: Execution Timeout Expired. ...
00:01:09 Request finished GET / - 500 0 - 12.8ms
00:01:10 Execution Dispatcher completed. <-- background DB query OK at same moment

Opus interpretation

A transient SQL connection interruption (possibly during the midnight task burst, or an Azure SQL idle/reconfiguration disconnect) leaves a connection in the web-request pool in a faulted state. ProGet keeps reusing that pool, so every subsequent web request fails instantly with "Execution Timeout Expired," while the separately-pooled background tasks are unaffected. A restart clears the pool and restores service.

Workarounds for now

• Added ConnectRetryCount=3;ConnectRetryInterval=10 to the SQL connection string.
• Azure App Service auto-heal: recycle the app on ≥50 HTTP 500s in 5 minutes.

If required I can provide more information or logs :)

Cheers
Carl

It was hard to boil this question down to a good heading but I got the question from a colleague if there is any way to view a package and see a list of all versions regardless of feed and in there have a column which lists the different feeds the versions exists in? Something like this that Azure Artifacts does

It would be a very nice feature for anyone seeking an overview of a single package and the ability to just at a glance see which of the versions has gone on to release

Cheers
Carl

@atripp Ahh ok, maybe that was what fixed it :)

Reverting to 25.0.9 did actually work in our case So there seems to be some issue with 25.0.10

I also got an error like this

Unhandled exception: Could not find stored procedure 'Executions_FailRunningExecutions'.

@viceice does your /health also show "no problems"?

Having some wierd behaviours with Proget after updating to 25.0.10 my /health responds 200 and shows that there are no issues, so no alerts are firing

{
"applicationName": "ProGet",
"databaseStatus": "OK",
"databaseStatusDetails": null,
"extensionsInstalled": {
"ProGet": "25.0.10",
"Inedo.SDK": "3.1.0",
"Amazon AWS": "3.1.3",
"InedoCore": "3.0.5",
"Azure": "3.1.2"
},
"licenseStatus": "OK",
"licenseStatusDetail": null,
"versionNumber": "2025.10 (Build 11)",
"releaseNumber": "25.0.10",
"startTime": 1758271289653,
"serviceStatus": "OK",
"serviceStatusDetail": null,
"replicationStatus": {
"serverStatus": null,
"serverError": null,
"clientStatus": null,
"clienterror": null,
"incoming": [],
"outgoing": []
}
}

But trying to go to the website I only get http error 500

The logs show an error "Could not find stored procedure 'Security_GetRoles'

2025-09-19T09:04:08.8094686Z info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2025-09-19T09:04:08.8095216Z       Request starting HTTP/1.1 GET http://proget.dev.companyname.com/health - - -
2025-09-19T09:04:08.8097717Z info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2025-09-19T09:04:08.8099365Z       Request finished HTTP/1.1 GET http://proget.dev.companyname.com/health - 200 619 application/json 0.3870ms
2025-09-19T09:04:10.6672063Z info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2025-09-19T09:04:10.6672953Z       Request starting HTTP/1.1 GET http://proget.dev.companyname.com/ - - -
2025-09-19T09:04:10.6687087Z An error occurred in the web application: Could not find stored procedure 'Security_GetRoles'.
2025-09-19T09:04:10.6778579Z info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2025-09-19T09:04:10.6779411Z       Request finished HTTP/1.1 GET http://proget.dev.companyname.com/ - 500 0 - 10.6553ms
2025-09-19T09:04:13.8057598Z info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2025-09-19T09:04:13.8058173Z       Request starting HTTP/1.1 GET http://proget.dev.companyname.com/health - - -
2025-09-19T09:04:13.8064277Z info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2025-09-19T09:04:13.8064479Z       Request finished HTTP/1.1 GET http://proget.dev.companyname.com/health - 200 619 application/json 0.4290ms
2025-09-19T09:04:13.9852857Z info: Microsoft.AspNetCore.Hosting.Diagnostics[1]

Anyone stumbled upon something similar and have any ideas on how to fix this?

Cheers
Carl

@apxltd I understand what you mean

When I have your attention I would like to just tell you that you have done an amazing work on ProGet as a product. It has everything we need while not being overcomplicated or overpriced. It really is a great product.

And the support through this forum is also always superb, fast and accurate.

Great work!

@apxltd I see, thanks for the info!

The use cases seems legit I think, like bundling multiple packages. Why would you like to have it undone?

I stumbled upon this V symbol when scrolling through Inedos open feeds. What does it mean and how does one add it? :)

@dean-houston

Ahh, thanks for pointing that out :)

As an answer to my question about health (and for anyone coming here in the future), I should've just read the docs properly, it was very clear what to check I love Proget! These things just exists and are well documented 🥰

https://docs.inedo.com/docs/proget-reference-api-health