Out of memory errors after upgrading to 2023.15

Valentijn

Hi,

We upgraded to Proget 2023.15 but now every morning the server is unavailable.

We see low virtual memory warnings in the eventlog mentioning the ProGet service.

ProGet.Service.exe (2812) consumed 10.343.374.848 bytes,

atripp

Hi @v-makkenze_6348 ,

Unfortunately this can be a really tricky issue to identify, as you'll need to figure out specifically what's causing these problems. It's often unrelated to ProGet, and could be caused by anything from Windows updates to low disk space.

I'll try to ask a few questions and give some tips on how to narrow things down.

The best place to start, what version did you upgrade from? If it was an earlier version of ProGet 2023, I would rollback; that will let you identify if it is in fact related to an upgrade.

The next thing I would try is disabling the ProGet Service; this is separate from the ProGet Web Service. The regular ProGet Service doesn't need to run for the Web Service (Web UI) to function.

If the problem goes away, then I restart the service but disable scheduled jobs (Admin > Scheduled Jobs) like feed clean up, vulnerability download, package analyzer, etc.

If the problem goes away, then I would try to find out which service specifically is causing problems.

You can also look at the execution logs (Admin > execution logs), and if something is taking a really long time, then that could be an indication of a problem

Thanks,
Alana

Valentijn

Hi

I upgrade whenever the website gives me a warning so the previous version was probably 2023.14 (can't find the previous version in the logfile)

It looks like the problem is with the feed cleanup as that takes a long time and gives an error.

Vicrea FeedCleanup 8/24/2023 1:43 AM - Normal Executing
Vicrea FeedCleanup 8/23/2023 1:58 AM 6h 32min Error Completed
Vicrea FeedCleanup 8/22/2023 1:49 AM 7h 29min Error Completed

The log doesn't say anything usefull:

DEBUG: 2023-08-22 23:58:38Z - Beginning cleanup for Vicrea (NuGet) feed.
INFO : 2023-08-22 23:58:38Z - Starting feed retention check...
DEBUG: 2023-08-22 23:58:38Z - Checking for feed retention rules...
DEBUG: 2023-08-22 23:58:38Z - Feed has 1 retention rule.
INFO : 2023-08-22 23:58:38Z - Running in dry run mode...
INFO : 2023-08-22 23:58:38Z - Checking rule 1...
DEBUG: 2023-08-22 23:58:38Z - Only delete packages that have not been requested in the last 90 days (since 5/25/2023 1:58:38 AM)
DEBUG: 2023-08-22 23:58:38Z - Only delete packages that have been downloaded fewer than 1 times.
DEBUG: 2023-08-22 23:58:38Z - Never delete the most recent 1 versions of packages.
INFO : 2023-08-22 23:58:38Z - Finding packages that match retention rule 1...

Valentijn

In todays log there is some more information:

Unhandled exception: Microsoft.Data.SqlClient.SqlException (0x80131904): There is insufficient system memory in resource pool 'internal' to run this query.
   at Microsoft.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at Microsoft.Data.SqlClient.SqlDataReader.TryHasMoreRows(Boolean& moreRows)
   at Microsoft.Data.SqlClient.SqlDataReader.TryReadInternal(Boolean setTimeout, Boolean& more)
   at Microsoft.Data.SqlClient.SqlDataReader.Read()
   at Inedo.Data.StrongDataReader.<Read>g__read|11_1[TRow](<>c__DisplayClass11_0`1& )
   at Inedo.Data.StrongDataReader.Read[TRow](IDbDataResult dbResult)+MoveNext()
   at Inedo.Data.StrongDataReader.Read[TRow](Func`1 getReader, Boolean disposeReader)+MoveNext()
   at Inedo.ProGet.Feeds.StandardPackageFeed`3.<>c__DisplayClass58_0.<<Inedo-ProGet-Feeds-IRetentionFeed<Inedo-ProGet-Feeds-StandardRetentionPackage<TVersion>>-EnumerateItemsAsync>g__iterateOneToOne|1>d.MoveNext()
   at Inedo.EnumerableExtensions.AsyncIterator`1.Enumerator.MoveNextAsync()
   at Inedo.ProGet.Feeds.IRetentionFeed`1.Inedo.ProGet.Feeds.IRetentionFeed.EnumerateItemsAsync(Boolean cachedOnly, Boolean prereleaseOnly, CancellationToken cancellationToken)+MoveNext()
   at Inedo.ProGet.Feeds.IRetentionFeed`1.Inedo.ProGet.Feeds.IRetentionFeed.EnumerateItemsAsync(Boolean cachedOnly, Boolean prereleaseOnly, CancellationToken cancellationToken)+MoveNext()
   at Inedo.ProGet.Feeds.IRetentionFeed`1.Inedo.ProGet.Feeds.IRetentionFeed.EnumerateItemsAsync(Boolean cachedOnly, Boolean prereleaseOnly, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
   at Inedo.ProGet.Feeds.RetentionRunner.RunRetentionRuleAsync(FeedRetentionRuleConfiguration rule, Int32 sequenceNumber, Boolean retentionDryRun, CancellationToken cancellationToken)
   at Inedo.ProGet.Feeds.RetentionRunner.RunRetentionRuleAsync(FeedRetentionRuleConfiguration rule, Int32 sequenceNumber, Boolean retentionDryRun, CancellationToken cancellationToken)
   at Inedo.ProGet.Feeds.RetentionRunner.PerformRetentionAsync(Boolean dryRun, CancellationToken cancellationToken)
   at Inedo.ProGet.ScheduledTasks.Feed.FeedCleanupScheduledTask.ExecuteAsync(ScheduledTaskContext context)
   at Inedo.ProGet.Service.Executions.ActiveScheduledTaskExecution.ExecuteAsync()
ClientConnectionId:9f725e0d-f135-46ae-8b92-fb58605389c0
Error Number:701,State:123,Class:17

atripp

Hi @v-makkenze_6348 ,

Thanks so much for narrowing that down! Thanks to that, I was able to find the issue and fix it.

This will be fixed in ProGet 2023.16 (we will release today or tomorrow), but I recommend to patch this.

To patch, download the SQL Script attached to PG-2466 and then run it against your ProGet database.

Cheers,
Alana

Valentijn

Thanks, I updated and feed cleanup is back to seconds instead of hours