I have Discovered Things and rather than just blabbing to Inedo support about them in an eternally-growing ticket chain in case I come up with useful information someday, I figured I might as well put it here on the chance it helps someone else too.
One of our use cases for Proget is to store local copies of... well, pretty much everything on an Ubuntu system, so that we can do reproducible builds from scratch. Some of our software vendors have a bad habit of taking down old packages from their .deb repos when a new version is published, which invariably breaks things at the worst possible times. So, lots of our engineers have asked "can we please just download all of Ubuntu and everything our 3rd party vendors provide?" While our head of IT says "there is no way we are paying for that much storage." So, I've set up Proget to act as a cache that just never clears itself, with connectors that pull .deb packages from the various upstream sources as they are used and keep local copies around in S3 buckets. All of Ubuntu is many terabytes of data and grows rapidly, but really we just need a few hundred gigs of packages that we use over and over again.
Proget is... not really designed for this use case. I can't really blame it. It deals pretty well with our 3rd party vendors that have 1000's of debian packages and add new updates a few times a month. But when I try to add make connectors that index all of the Ubuntu repos, including updates and security patches and stuff, it bogs down horribly. If you add .deb connectors with 100,000's of packages then it looks like it works once or twice, but sooner or later Something Happens which makes the feed unusable, and it then stays that way basically forever until you delete the connector entirely and re-create it. Inedo's been very helpful about fixing stuff where they can, but I'm abusing it into doing things it's not intended to do and redesigning the entire thing to scale better would take time.
We're running the Docker release of Proget 24.37 at the moment, and when it bogs down it tends to create error messages in the system logs like these:
Local index file update failure: database is locked
database is locked
code = Busy (5), message = System.Data.SQLite.SQLiteException (0x87AF00AA): database is locked
database is locked
at System.Data.SQLite.SQLite3.Step(SQLiteStatement stmt)
at System.Data.SQLite.SQLiteDataReader.NextResult()
at System.Data.SQLite.SQLiteDataReader..ctor(SQLiteCommand cmd, CommandBehavior behave)
at System.Data.SQLite.SQLiteCommand.ExecuteNonQuery(CommandBehavior behavior)
at System.Data.SQLite.SQLiteTransaction.Begin(Boolean deferredLock)
at System.Data.SQLite.SQLiteConnection.BeginDbTransaction(IsolationLevel isolationLevel)
at Inedo.ProGet.Feeds.CachedIndexFile.RequestExclusiveLock(String reason) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E561202\Src\ProGetCoreEx\Feeds\CachedIndexFile.cs:line 48
at Inedo.ProGet.Feeds.Debian2.Debian2Connector.UpdateIndexAsync(Debian2ConnectorIndex localIndex)
at Inedo.ProGet.Feeds.Debian2.Debian2Connector.Inedo.ProGet.Feeds.ILocalIndexFileConnector.UpdateIndexAsync(CancellationToken cancellationToken) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E561202\Src\ProGetCoreEx\Feeds\Debian2\Debian2Connector.cs:line 201
at Inedo.ProGet.Feeds.CachedIndexFileMan.UpdateConnectorIndexAsync(Int32 connectorId, CancellationToken cancellationToken) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E561202\Src\ProGetCoreEx\Feeds\CachedIndexFileMan.cs:line 53
::Connector Error on 06/18/2025 14:36:03::
URL: http://proget.example.com/feeds/deb_ubuntu_updates_upstream
Referrer: https://proget.example.com/feeds/deb_ubuntu_updates_upstream
User: ...
User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:137.0) Gecko/20100101 Firefox/137.0
Stack trace: at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1ContentLengthMessageBody.ReadAsyncInternal(CancellationToken cancellationToken)
at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
at Microsoft.AspNetCore.WebUtilities.FormPipeReader.ReadFormAsync(CancellationToken cancellationToken)
at Microsoft.AspNetCore.Http.Features.FormFeature.InnerReadFormAsync(CancellationToken cancellationToken)
at Microsoft.AspNetCore.Http.Features.FormFeature.ReadForm()
at Inedo.Web.AhHttpRequest.<.ctor>b__6_2()
at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
at System.Lazy`1.CreateValue()
at Inedo.Web.PageFree.SimplePageBase.ProcessControlTree()
at Inedo.Web.PageFree.SimplePageBase.ExecutePageLifeCycleAsync()
at Inedo.Web.PageFree.SimplePageBase.ProcessRequestAsync(AhHttpContext context)
at Inedo.Web.AhWebMiddleware.InvokeAsync(HttpContext context)
::Web Error on 07/16/2025 14:54:42::
An error occurred processing a GET request to http://proget.example.com/debian/deb_ubuntu_updates_upstream/dists/focal-updates/InRelease: An attempt was made to transition a task to a final state when it had already completed.
System.InvalidOperationException: An attempt was made to transition a task to a final state when it had already completed.
at System.Threading.Tasks.TaskCompletionSource.SetResult()
at Inedo.Threading.AhReaderWriterLock.ReleaseWrite() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E561202\Src\ProGetCoreEx\_ToInedoLib\AhReaderWriterLock.cs:line 114
at Inedo.ProGet.Feeds.Debian2.DebianIndexCache.GetReleaseAsync(Debian2Feed feed, String distro, CancellationToken cancellationToken) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E561202\Src\ProGetCoreEx\Feeds\Debian2\DebianIndexCache.cs:line 52
at Inedo.ProGet.WebApplication.FeedEndpoints.Debian2.DebianReleaseIndexHandler.ProcessRequestAsync(AhHttpContext context, WebApiContext apiContext, Debian2Feed feed, String relativeUrl) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E561202\Src\ProGet.WebApplication\FeedEndpoints\Debian2\DebianReleaseIndexHandler.cs:line 29
at Inedo.ProGet.WebApplication.FeedEndpoints.Debian2.Debian2FeedHandler.ProcessRequestAsync(AhHttpContext context, WebApiContext apiContext, Debian2Feed feed, String relativeUrl) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E561202\Src\ProGet.WebApplication\FeedEndpoints\Debian2\Debian2FeedHandler.cs:line 61
at Inedo.ProGet.WebApplication.FeedEndpoints.FeedEndpointHandler.FeedRequestHandler.ProcessRequestAsync(AhHttpContext context) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E561202\Src\ProGet.WebApplication\FeedEndpoints\FeedEndpointHandler.cs:line 198
::Web Error on 07/16/2025 11:23:12::
We had this limping along for a little while, but then Proget also outright crashed a couple times with the somewhat concerning message Bus error, and no backtrace or anything like that. Linux's Bus error message normally means "bad memory or motherboard" which is something I'd hope isn't likely to happen on an AWS VM, so we were kind of mystified. However, it turns out that Linux will also give you Bus error when a program asks for shared memory and for whatever reason it can't give it any. What's going to use a lot of shared memory? Well... a docker container like we're running Proget in, or maybe SQLite which Proget uses for storing connector cache info. Trying to update these massive connectors was just straight up OOM'ing the system sometimes, in a slightly bass-ackwards way. When it happened most of the time the request would time out, the thread would get killed, and the memory would be freed again so we never saw it. (Better monitoring is on our to-do list.) Turned out that similar things were also happening with disk space: the SQLite database that stored cached package metadata for the connector would explode to 80+ gigabytes, run the server out of disk space, and then something would crash and the database would get cleaned back up to a reasonable size. But something remained slightly wedged in the DB and so after that point whenever the connector tried to fetch package lists from upstream, it would (slowly) explode the RAM or disk usage and cause the same error all over again. Only way to fix it seemed to be to entirely delete and recreate the connector to the giant repo, which would coax it into working another few times before the problem happened again.
So we have now acquired 512 GB of disk space and 32 GB of RAM, and now our ridiculous giant connectors work! They don't work well, they're laggy and slow and I needed to buy more I/O bandwidth for our poor AWS server, but it's usable for now. I'll probably keep working on it and tuning; upgrading to Proget 2025 is on the to-do list in a month or two.
So hopefully this is useful to someone at some point, and I'll keep sharing my adventures in software abuse as I go.
Ref: [EDO-11905]