Errors in PackageAnalyzer logs

v.makkenze_6348

Hi,

I see errors in PackageAnalyzer logs

Unhandled exception analyzing packages in Vicrea feed: Object reference not set to an instance of an object.
Unhandled exception analyzing packages in VicreaMaven feed: Object reference not set to an instance of an object.
Unhandled exception: Microsoft.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.
 ---> System.ComponentModel.Win32Exception (258): The wait operation timed out.
   at Microsoft.Data.SqlClient.SqlCommand.<>c.<ExecuteDbDataReaderAsync>b__209_0(Task`1 result)
   at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
   at Inedo.Data.DatabaseContext.DbResult.CreateAsync(DbCommand command, DatabaseContext context, DateTimeOffset startTime, Stopwatch stopwatch)
   at Inedo.Data.DatabaseContext.ExecuteInternalAsync(String storedProcName, GenericDbParameter[] parameters)
   at Inedo.Data.SqlServerDatabaseContext.ExecuteInternalAsync(String storedProcName, GenericDbParameter[] parameters)
   at Inedo.ProGet.Data.DB.Context.Projects_GetReleaseAnalyzerDataAsync(Nullable`1 ProjectRelease_Id)
   at Inedo.ProGet.Projects.ProjectReleaseAnalyzer.AnalyzeReleaseAsync(Int32 projectReleaseId, Boolean sendEmail, CancellationToken cancellationToken)
   at Inedo.ProGet.ScheduledTasks.General.PackageAnalyzerScheduledTask.AnalyzeReleasesAsync(ScheduledTaskContext context)
   at Inedo.ProGet.ScheduledTasks.General.PackageAnalyzerScheduledTask.ExecuteAsync(ScheduledTaskContext context)
   at Inedo.ProGet.Service.Executions.ActiveScheduledTaskExecution.ExecuteAsync()
ClientConnectionId:0d07b9a9-043b-4747-81e1-0bdfc00192d0
Error Number:-2,State:0,Class:11

After the last error the analysis stops.

Using Version 2023.32 (Build 3)

v.makkenze_6348

After deleting a lot of SBOM's and running sp_updatestats on the database the timeout is gone (for now) but the Object reference errors are still there.

atripp

Hi @v-makkenze_6348 ,

This was all totally rewrite in ProGet 2024 (releasing in a few weeks), so we don't anticipate that we'll see these errors.

If you want to send us a database backup, we'd be happy to use it in our testing, and make sure it works after the ProGet 2024 upgrade. Reduced risk for you, more testing and real-world data for us :)

Anyway, if you want to do that, please submit a ticket so that we can send you a secure link to upload the database BAK file.

Cheers,
Alana

v.makkenze_6348

I upgraded to 2024 and now I have these errors during analysis

ERROR: 2024-04-21 14:38:08Z - Unhandled exception analyzing pkg:nuget/VicreaNeuronGegevensmagazijnContracts pkg:nuget/VicreaNeuronGegevensmagazijnContracts@17.2.65 in Vicrea feed: 547`16`0`FeedPackageVersions_SetAnalysis`26`The INSERT statement conflicted with the FOREIGN KEY constraint "FK__FeedPackageAnalyses__FeedPackageVersions". The conflict occurred in database "ProGet", table "dbo.FeedPackageVersions".

rhessinger

Hi @v-makkenze_6348,

As @atripp stated in your other post, this is due to bad data. For that package exact;y, it was added with a NuGet quirks version that is 4 parts (most likely specified), 17.2.65.0, which is getting handled correctly to a 3 part version due to NuGet's API specs. We are still working out how best to handle these cases.

Thanks,
Rich

atripp

Hi @v-makkenze_6348 ,

Just to give you an update, we're still working on this; there's a lot of duplicated data and figuring out how to clean it up is a challenge. It's hard to say where it came from.

For example, your database has internal ids for both bootstrap.less-3.4.1 and Bootstrap.Less-3.4.1, and we need to figure out how safely merge those. At this point, we're thinking we're just going to do one massive SQL Script to do a cleanup. Then, we can try to figure out where the duplicates are coming from.

Regarding VicreaNeuronGegevensmagazijnContracts package, we can't find that in your database. But I assume that's a new package/version. If you have a newer backup, please upload that to the link we sent - we can check it in there. I think its' related to 4-part version, but just want to check.

Cheers,
Alana

v.makkenze_6348

VicreaNeuronGegevensmagazijnContracts indeed had a four part version number.
We deleted al versions of the package as the last download was in 2018.

v.makkenze_6348

Strange that you can't see VicreaNeuronGegevensmagazijnContracts, I can't find bootstrap.less

atripp

Thanks for clarifying, that's good to know taht solved the problem! I looked again and found it. Maybe I was looking in the wrong database

It doesn't look like bootstrap.less is in your feeds, but it is in the PackageNameIds table:

Just because it's in that table doesn't mean you're using it; we use the PackageName_Id as a surrogate ID instead of the package name. There should only be one entry, since NuGet package names are case insensitive.

anyway, I think I found the culprit -- looks like it was added during a vulnerability database update. MAybe there is some bug that didn't properly check the casing

Anyways, that's what we have to clean-up up :)

stevedennis

Hi @v-makkenze_6348 ,

Just an FYI, we do have a duplicates clean-up script available, and ran it against your database with no issues.

https://gist.github.com/apxltd/351d328023c1c32852c30c335952fabb

We haven't yet decided how to roll it out, but if you do run into more package analyzer issues like this (particularly on the SBOM side), it might help.

Thank you,
Steve

v.makkenze_6348

Thanks I made a backup and ran the script and it did clean up quite a few rows.
I'll try upgrading to 2024 again next week after we released :)