Timeout in build when running pgscan

v.makkenze_6348

After upgrading to the latest version this started to happen.

Version: Version 2023.30 (Build 16)

Server responded with 500 Internal Server Error: Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.

In Diagnostic Center the following is shown:

Unhandled exception processing https://packages.vicrea.nl/api/sca/import

Microsoft.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
---> System.ComponentModel.Win32Exception (258): The wait operation timed out.
at Microsoft.Data.SqlClient.SqlCommand.<>c.<ExecuteDbDataReaderAsync>b__209_0(Task`1 result)
at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
at Inedo.Data.DatabaseContext.DbResult.CreateAsync(DbCommand command, DatabaseContext context, DateTimeOffset startTime, Stopwatch stopwatch)
at Inedo.Data.DatabaseContext.ExecuteInternalAsync(String storedProcName, GenericDbParameter[] parameters)
at Inedo.Data.SqlServerDatabaseContext.ExecuteInternalAsync(String storedProcName, GenericDbParameter[] parameters)
at Inedo.ProGet.Data.DB.Context.Projects_GetReleaseAnalyzerDataAsync(Nullable`1 ProjectRelease_Id)
at Inedo.ProGet.Projects.ProjectReleaseAnalyzer.AnalyzeReleaseAsync(Int32 projectReleaseId, Boolean sendEmail, CancellationToken cancellationToken)
at Inedo.ProGet.Projects.BomUtil.ImportXmlAsync(Stream source)
at Inedo.ProGet.WebApplication.SimpleHandlers.Api.Bom.ScaApiHandler.ProcessRequestAsync(AhHttpContext context, LoggedResponseStream output, WebApiContext apiContext)
at Inedo.ProGet.WebApplication.SimpleHandlers.Api.ProGetApiKeySecuredHandler.ProcessRequestInternalAsync(AhHttpContext context, LoggedResponseStream output, WrappedApiKey apiKey)
at Inedo.Web.Handlers.Api.ApiKeySecuredHandler.ProcessRequestInternalAsync(AhHttpContext context)
at Inedo.Web.Handlers.Api.StandardApiHandler.ProcessRequestAsync(AhHttpContext context)
ClientConnectionId:6200fa0a-d298-4898-be14-9721cbe745c5
Error Number:-2,State:0,Class:11

atripp

Hi @v-makkenze_6348 ,

I'm not sure what version you upgraded from, but there hasn't really been many changes that would have caused this I think.

This is a general database timeout error, which could be happening under heavy load (so trying again should work), or it could be a result of needing to update statistics, etc.

Thanks,
Alana

v.makkenze_6348

I ended up restarting the build 5 times when it finally worked.
Proget is running on a dedicated machine that does nothing else and no other builds where running.

Can it be that we have too many SBOMs, versions, packages or something like that?

atripp

It's really hard to say, but that's probably it. We'd really need to analyze it further to tell. How many packages are in the release?

FYI - this is all getting a "total rewrite" in ProGet 2024, and you can preview some of the features in the latest release of PRoGet

v.makkenze_6348

Running again into this issue:
Unhandled exception processing https://packages.vicrea.nl/api/sca/import

Version 2023.31 (Build 5)
Total Packages 635

I deleted the release in Proget and build again, again a timeout but I do see the new release in Proget. So it looks like the SBOM is processed but maybe not complete?

v.makkenze_6348

I noticed that the new version had multiple SBOM's and I only ran the build once.
After deleting all SBOM's and the version again the build still failed but at least I only see 1 SBOM this time.

Edit: The build runs pgscan twice once for npm and once for NuGet so i should see two SBOM's
499 components in npm SBOM
137 components in NuGet SBOM
636 is the expected number of packages

But this time total packages in UI shows 634

atripp

Hi @v-makkenze_6348,

Just want to confirm here -- it sounds like it's working, except the issue now that you're only seeing 634 packages but not 363?

One thing I suspect is that one of the packageurls in the SBOM is invalid (incorrect format). Are you able to find which item isn't being imported?

It might be missing a version or something else?

Thanks,
Alana

v.makkenze_6348

Hi,

I checked again and the number of packages is correct when the build succeeds.
Had to run the build 9 times before I had a build that did not give a timeout.

Before i counted the number of components in the SBOM but the app itself is also a component and i have two SBOM's so therefore i was missing two packages.

I get the feeling that during the failing builds the SBOM is uploaded but not completely processed. So the next build its processed and thats when i see three SBOM's or four if the build succeeds.

Still no clue why this is the only build that has these failures, its not our biggest or most complex product. We don't have many builds that contain both npm and nuget packages but there are others. All builds are run at least once a week so the others with two pgscans are ok.