Proget Service/Site Crashes with 500 Error

jfullmer_7346

===What version does this affect?===
ProGet 2023.13 (Inedo Hub)

===How can we help? (in brief)===
[Major] Service keeps crashing taking site down every few days

===Please add any details that might help us help you.===
On a fairly regular basis our proget site is crashing and giving a 500 error. It's often happening in the wee hours of the morning.
Sometimes it's fixed by a restart of just the proget service
Sometimes it's a restart of the service and the site
Sometimes it's a restart of the whole iis server service

We're also able to bring it back up but it shouldn't be randomly crashing.

When checking in the event log we see a few things and I am attaching saved versions of those logs so you can see them in more detail.
Around the same time as things crash there are winrm and certificate related errors (we're not doing anything special that we know of with certificates and winrm) and we see an error saying the proget service couldn't connect to the sql server

jfullmer_7346

I can't attach a zip of the events so here are some screenshots

IIS AspNetCoreModuleV2 Error Code 1018

IIS AspNetCoreModuleV2 Error Code 1007

CertificateServicesClient-CertEnroll Error 86

.NET Runtime Error 1026

Service Control Manager Error 7031

stevedennis

Hi @jfullmer_7346 ,

The ProGet Service (and WebApp if using IIS) will crash when the database is no longer accessible. Based on the error messages, that exactly the case. The "good news" isn't that isn't ProGet related, so that at least gives you one less place to look.

It looks like you're using the SQL Server on the same machine ("shared memory provider"), but I'm not totally sure. If that's the case, then my guess is that the SQL Server is crashing; you'd have to check SQL Server's event/error logs for that. It's very rare for SQL Server to crash, and I'd be worried that it's a sign of hardware failure.

Beyond that, I don't have any specific trips/ticks on research SQL Server connectivity problems, but but if you search ...

"shared memory provider error 40" (it was in the error message I saw)
"SQL Server v??? crashing" (assuming this is the case, whatever your version is)
specific error messages from SQL server's event/error logs about the time it crashed

... you'll find lots of advice all over the place, since this could impact pretty much any software that uses SQL Server.

Good luck and let us know what you find!

Thanks,
Steve

jfullmer_7346

@stevedennis said in Proget Service/Site Crashes with 500 Error:

shared memory provider error 40

Thank you for the insights, I was also thinking the sql server was the most significant thing. It is on the same server but it's also a virtual server, I don't think it's a hardware failure there.

There is another sql server running on the server for chocolatey central management, there may have been a conflict with a mismatched version dependency on that app for the dotnet-aspnetmodule-v2 where I need the 16.x version for CCM but I accidentally upgraded to the 17.x version.

At the moment the issues hasn't recurred since the day I reported it and I'm hoping having fixed that dependency has fixed the issue. I found some other things that mention local sql aliases needing to exist, I do use many dns aliases in the server's config so I might try making those as sql named pipe aliases to see if that makes any difference, though it seems a long shot.