ProGet hangs after several queries/restores, leaves HTTP error in system event log

We have a ProGet server that we're running as a Windows Server 2012R2 VM. The database is Azure SQL. We're using IIS as the web host.

We were running 4.0.9 and we needed to join the machine to a domain (a newly created Active Directory). After joining the domain, we found that the ProGet service would "hang" after being queried for a while. A great example is that we would try to run a build of one of our products that requires internal packages from ProGet. The build could restore packages for three or four projects/assemblies, but then timeouts would start.

In the Application error log I see this error:

Event sequence: 3 
Event occurrence: 2 
Event detail code: 0 
 
Application information: 
    Application domain: /LM/W3SVC/2/ROOT-2-131122237961467438 
    Trust level: Full 
    Application Virtual Path: / 
    Application Path: C:\Program Files\ProGet\WebApp\ 
    Machine name: NUGET 
 
Process information: 
    Process ID: 680 
    Process name: w3wp.exe 
    Account name: IIS APPPOOL\ProGet 
 
Exception information: 
    Exception type: HttpException 
    Exception message: Server cannot set status after HTTP headers have been sent.
   at System.Web.HttpResponse.set_StatusCode(Int32 value)
   at Inedo.ProGet.WebApplication.SimpleHandlers.NuGet.ODataHandler.WriteError(HttpContext context, Int32 statusCode, String message, String fullMessage)
   at Inedo.ProGet.WebApplication.SimpleHandlers.NuGet.ODataHandler.<ProcessRequestAsync>d__11.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.TaskAsyncHelper.EndTask(IAsyncResult ar)
   at System.Web.HttpApplication.CallHandlerExecutionStep.OnAsyncHandlerCompletion(IAsyncResult ar)

Immediately after that, I see IIS present an error in the Application log:

ISAPI 'C:\Windows\Microsoft.NET\Framework64\v4.0.30319\aspnet_isapi.dll' reported itself as unhealthy for the following reason: 'Deadlock detected'.

Once the ProGet server eventually comes back up (or I manually recycle the service/IIS app) I see this in the ProGet log:

URL: https://nuget.silverlining.ventures/nuget/Default/FindPackagesById()?id='Desktop.Analyzers'
Referrer: (not set)
User: team.city
User Agent: NuGet xplat/3.5.0-beta2-1484 (Microsoft Windows 6.3.9600 )
Stack trace:    at System.Web.HttpResponse.set_StatusCode(Int32 value)
   at Inedo.ProGet.WebApplication.SimpleHandlers.NuGet.ODataHandler.WriteError(HttpContext context, Int32 statusCode, String message, String fullMessage)
   at Inedo.ProGet.WebApplication.SimpleHandlers.NuGet.ODataHandler.<ProcessRequestAsync>d__11.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.TaskAsyncHelper.EndTask(IAsyncResult ar)
   at System.Web.HttpApplication.CallHandlerExecutionStep.OnAsyncHandlerCompletion(IAsyncResult ar)

Basically the same error as seen in the Application event log.

Is there somewhere else I should look for additional details? My initial guess by looking at this is that the real problem is being masked by this error.

Other things that may help:

We were using Azure Active Directory domain services and switched our AD to be a "real" AD on a real Windows Server 2012 R2 image.
We are synchronizing our AD with Azure Active Directory. (I noticed there was a release note earlier about a fix for querying invalid LDAP usernames or something; not sure if this overlaps with that.)

Product: ProGet
Version: 4.1.3

You are correct; this is an error message in displaying the error message. But the underlying error is most likely that a connection was suddenly dropped or that a system resource (database, network, etc) is unavailable.

This is especially common during a package restore because NuGet is effectively doing a Denial of Service attack against your server by doing hundreds (sometimes thousands) of expensive, simultaneous queries during restore.

The ways to mitigate are:

remove connectors to nuget.org
load balance the instance
lessen number of packages used by solution
upgrade or downgrade nuget.exe for better performance
increase the Web.ODataOutputCachePeriod value in Advanced Settings

ProGet 4.1 has some async handling that may help, but if the problem is intermediate hardware that can't handle the load (as is often the case), then load balancing is about the only option.

We don't have any connectors to nuget.org.

We're building .NET Core projects so, while we have a minimum number of packages to restore, those packages restore additional chained-in packages and we can't reduce those.

We are using the latest nuget.exe for solution-level packages, but we have to use the dotnet CLI for other package restore operations so we don't have control over it.

Increasing Web.ODataOutputCachePeriod to something greater than zero results in package restore failures with the dotnet CLI:

Restoring packages for D:\tc\w\855be35b5addb7b9\src\MyProject\project.json...
 Retrying 'FindPackagesByIdAsyncCore' for source 'https://mynugetserver/nuget/Default/FindPackagesById()?id='Desktop.Analyzers''.
 The content at 'https://mynugetserver/nuget/Default/FindPackagesById()?id='Desktop.Analyzers'' is not valid XML.
   Data at the root level is invalid. Line 1, position 1.

That happens for every package that the dotnet CLI tries to restore. We are currently trying with their latest version - 1.0.0-preview3-003171.

The database is Azure SQL so I'm hoping that it's not overwhelmed by a few NuGet queries.

The VM is an Azure VM but may be under-sized. I don't notice any CPU or memory spikes when the package restore is occurring but I will try increasing the size anyway. Right now it's a "Standard D1 - 1 core, 3.5GB RAM." I'll try doubling that to two cores, 7GB RAM.

After upgrading both the ProGet VM and our (potentially underpowered) Active Directory VM to more cores/more RAM, we seem to have worked around the issue. Thanks for the tips!