Welcome to the Inedo Forums! Check out the Forums Guide for help getting started.
If you are experiencing any issues with the forum software, please visit the Contact Form on our website and let us know!
Proget 2023.7 deadlocks on Get for Cached Packages
-
Microsoft.Data.SqlClient.SqlException (0x80131904): Transaction (Process ID 132) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
at Microsoft.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction) at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose) at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady) at Microsoft.Data.SqlClient.SqlDataReader.TryHasMoreRows(Boolean& moreRows) at Microsoft.Data.SqlClient.SqlDataReader.TryReadInternal(Boolean setTimeout, Boolean& more) at Microsoft.Data.SqlClient.SqlDataReader.ReadAsyncExecute(Task task, Object state) at Microsoft.Data.SqlClient.SqlDataReader.InvokeAsyncCall[T](SqlDataReaderBaseAsyncCallContext1 context)
--- End of stack trace from previous location ---
at Inedo.Data.StrongDataReader.ReadAllAsync[TRow](IDbDataResult dbResult)
at Inedo.Data.DatabaseContext.ExecuteTableAsync[TRow](String storedProcName, GenericDbParameter[] parameters)
at Inedo.Data.DatabaseContext.ExecuteTableAsync[TRow](String storedProcName, GenericDbParameter[] parameters)
at Inedo.ProGet.Feeds.NuGet.NuGetFeed.GetPackageAsync(String id, NuGetVersion version)
at Inedo.ProGet.WebApplication.FeedEndpoints.NuGet.NuGetApi.GetHandler.ProcessGetRequestAsync(AhHttpContext context, WebApiContext apiContext, RequestData urlData, NuGetFeed feed)
at Inedo.ProGet.WebApplication.FeedEndpoints.NuGet.NuGetApi.ProcessRequestAsync(AhHttpContext context, WebApiContext apiContext, NuGetFeed feed, String relativeUrl)
at Inedo.ProGet.WebApplication.FeedEndpoints.NuGet.NuGetFeedHandler.ProcessRequestAsync(AhHttpContext context, WebApiContext apiContext, NuGetFeed feed, String relativeUrl)
at Inedo.ProGet.WebApplication.FeedEndpoints.FeedEndpointHandler.FeedRequestHandler.ProcessRequestAsync(AhHttpContext context)
ClientConnectionId:f2521b73-0e45-4796-bb52-faac892ad32f
Error Number:1205,State:51,Class:13
-
Hi @chuck-buford_5284 ,
I'm surprised to see these on v2023.7, but it's an issue we're working through (it's an entirely new indexing system).
Can you provide us with your deadlock reports?
It should be on your SQL Server, under Management > Extended Events > SEssions > system_health > package0.event_file. Then you can click on Filter (or CTRL-R), and add a filter for Field
name=xml_deadlock_report.Ultimately what we're looking for are the xml files, specifically what two queries are deadocking.
Here is some screenshots on how to do that:
https://www.mssqltips.com/sqlservertip/6430/monitor-deadlocks-in-sql-server-with-systemhealth-extended-events/Cheers,
Alana
-
I'm not seeing how to upload the files. (And the plain text xml is too long to post itself.
The xml file (for one deadlock) had seven processes in it.
1-6 were:
SELECT * FROM [NuGetFeedPackageVersions_Extended] WHERE [Feed_Id] = @Feed_Id AND
[Package_Name_Lower] = LOWER(@Package_Id) AND (@Version_Text IS NULL OR
[Package_Version] = @Version_Text#7 (And I suspect the deadlocker) was:
UPDATE [Feeds] SET [FeedState_Number] = COALESCE([FeedState_Number], 0) + 1
WHERE [Feed_Id] = @Feed_I
-
Hi @chuck-buford_5284 ,
Thanks; that's exactly what I would have looked for in the file, thanks for sending that;
How often are these coming up? What sort of hardware are you working with? Are you able to reproduce this consistently?
The query pattern is implying that there's heavy usage while uploading or deleting packages, and I know we spotted some potential issues earlier - but wanted to wait to confirm something else.
We have some optimized versions of
FeedPackageVersions_DeletePackageVersionandFeedPackageVersions_CreateOrUpdatePackageVersion, but we didn't ship them just yet. Can you try them out? Just run the attached queries in https://inedo.myjetbrains.com/youtrack/issue/PG-2387Cheers,
Alana
-
Able to reproduce consistently (Brand new server. Transferred all internal nuget packages from old server, and any large build pulling cached, or caching external, nuget packages)
VM
Windows Server 2019 Standard 64 Bit
32 GB Ram
Intel(R) Xeon(R) CPU E5-4650 v3 @ 2.10GHz (2 Processors)I will try out the attached queries and report back the results.
-
I ran it with the new stored procs, and I still got deadlocks, although far far fewer.
Looking at the xml, there was only 1 query deadlocked by the update instead of 6.
However, re-running the build (and the nuget restore, obviously) was able to resolve the deadlocks (I assume this is because it's deadlocking when caching packages, and with fewer deadlocking, more packages were cached).
So, good news so far. It's a definite improvement.
Now I just have to track down why 2 packages will restore directly from nuget, but not through proget's connector.... but that's a different issue.
-
Hi @chuck-buford_5284 ,
Thanks for letting me know; can you let me know what queries were deadlocked? We shouldn't see any deadlock, but I guess it'd be possible for
SELECT * FROM [NuGetFeedPackageVersions_Extended]to deadlock on itself depending on the query plan sql server uses.There are a few other things we can try, but we can't repro this at all, even in a test lab that's just hammering the database. Are you using SQL Server Express (i.e. what Inedo Hub installs by default)? It should work the same of course...
Caching packages use that "CreatePackage method", so it's basically the same thing as installing a package I suppose.
Cheers,
Alana
