I took a while to get back to this thread, but here goes:
Today we upgraded to the MSSQL backed version of ProGet, that is currently on the Docker hub (5.2.9.11), and most of it went smooth.
However, when restoring all the packages, I did get the unresponsive errors again. But this time the server came back and we could see the log messages. After changing the connection string, things seem to be more stable now when doing a full restore.
The new connection string has this appended to it: "Max Pool Size=200;"
Hopefully this version keeps us happier, as the old version needs a reboot every other day.