Occasional error when downloading a package

We have ProGet deployed and are occasionally getting errors when packages are being downloaded, especially if multiple members of my team are restoring packages for a solution at the same time. We've hosted ProGet in IIS, so I have enabled Failed Request Tracing and noticed that a NullReferenceException occurs. Any clues on how we might fix this? We haven't been able to detect any particular pattern in when we get these errors, although it seems to occur more often when more members of my team are downloading packages at the same time.

I've uploaded the Failed Request Trace here: https://dl.dropboxusercontent.com/u/23379493/FailedRequestTrace.zip

Product: ProGet
Version: 3.7.6

That's strange... Is anything getting written to ProGet's error log (accessible by clicking on the Admin->view all messages link)? Unhandled exceptions in the web application like that should be written there with a complete stack trace. That might help us figure out the root cause.

I had a look at the errors being reported in the admin section. Here's what I found:

Logged:

8/20/2015 9:34:08 AM

Level:

Warning Warning

Category:

NuGet API

Message:

An error occurred processing a GET request to http://zeusapp002.zeus.corp:9090/nuget/Developers/package/DeVries.NServiceBus/1.0.0: Object reference not set to an instance of an object.

Details:

at Inedo.ProGet.WebApplication.SimpleHandlers.NuGet.NuGetApi.GetHandler.ProcessGetRequest(HttpContext context, RequestData urlData)
at Inedo.ProGet.WebApplication.SimpleHandlers.NuGet.NuGetApi.ProcessRequest(HttpContext context)

It looks like some information is missing from the stack trace; a full stack trace should be logged. Can you find that?

Any clues as to where I might find that log? I've looked into the installation directory but couldn't find any log files.

It'd be in the admin section errors; there should be a full stack trace.

All that came through is this...

at Inedo.ProGet.WebApplication.SimpleHandlers.NuGet.NuGetApi.GetHandler.ProcessGetRequest(HttpContext context, RequestData urlData) at Inedo.ProGet.WebApplication.SimpleHandlers.NuGet.NuGetApi.ProcessRequest(HttpContext context)

.. was there anything else?

Nope, that's all I got. I did notice a new error I haven't seen before just now:

Logged:

8/21/2015 2:45:36 PM

Level:

Warning Warning

Category:

NuGet API

Message:

An error occurred processing a GET request to http://zeusapp002.zeus.corp:9090/nuget/Developers/package/DeVries.WMS.Database.Implementation/1.0.0: An error occurred while communicating with the remote host. The error code is 0x800703E3.

Details:

at System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6.FlushCore(Byte[] status, Byte[] header, Int32 keepConnected, Int32 totalBodySize, Int32 numBodyFragments, IntPtr[] bodyFragments, Int32[] bodyFragmentLengths, Int32 doneWithSession, Int32 finalStatus, Boolean& async)
at System.Web.Hosting.ISAPIWorkerRequest.FlushCachedResponse(Boolean isFinal)
at System.Web.HttpResponse.Flush(Boolean finalFlush, Boolean async)
at System.Web.HttpWriter.WriteFromStream(Byte[] data, Int32 offset, Int32 size)
at Inedo.Web.Handlers.HttpResponseExtensions.TransmitStream(HttpResponse response, Stream stream, String fileName, String contentType)
at Inedo.ProGet.WebApplication.SimpleHandlers.NuGet.NuGetApi.GetHandler.TransmitPackage(HttpContext context, ProGetFeedPackage package, NuGetFeeds_Extended feed, String packageFileName, Boolean includeSymbols, NuGetFeed2 feed2)
at Inedo.ProGet.WebApplication.SimpleHandlers.NuGet.NuGetApi.GetHandler.ProcessGetRequest(HttpContext context, RequestData urlData)
at Inedo.ProGet.WebApplication.SimpleHandlers.NuGet.NuGetApi.ProcessRequest(HttpContext context)

Any updates on this issue? I'm still seeing these messages being logged with 3.8.0.

Logged:

9/10/2015 1:35:02 PM

Level:

Warning Warning

Category:

NuGet API

Message:

An error occurred processing a GET request to http://zeusapp002.zeus.corp:9090/nuget/Developers/package/DeVries.WMS.ICSecurity.Headers/1.0.0: Object reference not set to an instance of an object.

Details:

at Inedo.ProGet.WebApplication.SimpleHandlers.NuGet.NuGetApi.GetHandler.ProcessGetRequest(HttpContext context, RequestData urlData)
at Inedo.ProGet.WebApplication.SimpleHandlers.NuGet.NuGetApi.ProcessRequest(HttpContext context)

I suspect the issue is related to network connectivity, based on stack traces messages (" An error occurred while communicating with the remote host"), and especially b/c it's intermittent and works after a retry.

ProGet is fairly well tested for high-volume usage. Can you come up with a way to consistently reproduce this? A script that does a lot of simultaneous downloads might help point in the right direction.

We've finally tracked down what was causing this issue. We had two feeds configured that both had two connectors that connected back to the same ProGet server on different feeds. Turns out that this causes a very high request queue on the ProGet server, which in turn caused timeouts and eventually errors during package restore.

I've now created a separate web application in IIS hosting the same ProGet folder but running on a different port and configured those two feeds to connect to the other port. This seems to alleviate the problem for now.

Would be nice though if this was supported by ProGet itself. I've seen many artifact repositories that support such a feature where sort of virtualized feeds can be created by combining other feeds.