Nuget Package Manager lists no packages under 'Online' tab because Proget throws error

Symptom: In Visual Studio 2013, with the latest Nuget Package Manager installed, if we click on the 'Online' tab, the spinner keeps spinning but no packages are shown. If we eventually cancel the request, an error message saying that 'the nuget server did not respond' is shown.
The 'Installed' and 'Updates' tab of the package manager work fine and list the nuget packages as expected.
This behavior is seen with multiple version of the Proget 3.4.x family. We are going to test with the older versions of the ProGet 3.3 family.

More Information: Using WireShark, we logged the communication between the machines.

Request: (from Nuget Package Manager)

GET http://<PROGET_SERVER>/nuget/<FEED_NAME>/Search()/$count?$filter=IsLatestVersion&searchTerm=''&targetFramework='net40%7Cnet40%7Cnet40%7Cnet40%7Cnet40%7Cnet40%7Cnet40'&includePrerelease=false HTTP/1.1
DataServiceVersion: 2.0;NetFx
MaxDataServiceVersion: 2.0;NetFx
User-Agent: NuGet VS Packages Dialog - Solution/2.8.50926.663 (Microsoft Windows NT 6.1.7601 Service Pack 1, VS Professional/12.0)
Accept-Charset: UTF-8
Accept: text/plain
Host: <PROGET_SERVER>
Accept-Encoding: gzip, deflate

Response: (from ProGet server)

HTTP/1.1 500 Internal Server Error
Cache-Control: private
Content-Type: text/html
Server: Microsoft-IIS/7.5
X-AspNet-Version: 4.0.30319
X-UA-Compatible: IE=edge
X-ProGet-Version: 3.4.7.40
X-Powered-By: ASP.NET
Date: Wed, 18 Feb 2015 18:37:17 GMT
Content-Length: 1208
Age: 2
Via: 1.1 localhost.localdomain

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"/>
<title>500 - Internal server error.</title>
<style type="text/css">
<!--
body{margin:0;font-size:.7em;font-family:Verdana, Arial, Helvetica, sans-serif;background:#EEEEEE;}
fieldset{padding:0 15px 10px 15px;} 
h1{font-size:2.4em;margin:0;color:#FFF;}
h2{font-size:1.7em;margin:0;color:#CC0000;} 
h3{font-size:1.2em;margin:10px 0 0 0;color:#000000;} 
#header{width:96%;margin:0 0 0 0;padding:6px 2% 6px 2%;font-family:"trebuchet MS", Verdana, sans-serif;color:#FFF;
background-color:#555555;}
#content{margin:0 0 0 2%;position:relative;}
.content-container{background:#FFF;width:96%;margin-top:8px;padding:10px;position:relative;}
-->
</style>
</head>
<body>
<div id="header"><h1>Server Error</h1></div>
<div id="content">
 <div class="content-container"><fieldset>
  <h2>500 - Internal server error.</h2>
  <h3>There is a problem with the resource you are looking for, and it cannot be displayed.</h3>
 </fieldset></div>
</div>
</body>
</html>

Looking at the event log on the Proget machine, we see the following exception log:

Event code: 3005 
Event message: An unhandled exception has occurred. 
Event ID: 10dcc61f2eef47faae4b0bc14a913c3b 
Event sequence: 27 
Event occurrence: 26 
Event detail code: 0 

Application information: 
    Application domain: <DOMAIN>
    Trust level: Full 
    Application Virtual Path: / 
    Application Path: C:\Program Files\ProGet\WebApp\ 
    Machine name: <PROGET SERVER NAME>

Process information: 
    Process ID: 20536 
    Process name: w3wp.exe 
    Account name: <USER ACCOUNT>

Exception information: 
    Exception type: InvalidOperationException 
    Exception message: Nullable object must have a value.
   at System.ThrowHelper.ThrowInvalidOperationException(ExceptionResource resource)
   at Inedo.ProGet.WebApplication.SimpleHandlers.NuGet.ODataHandler.ProcessVirtualRequest(ODataResponseContext context)
   at Inedo.ProGet.WebApplication.SimpleHandlers.NuGet.ODataHandler.ProcessRequest(HttpContext context)
   at Inedo.ProGet.WebApplication.SimpleHandlers.NuGet.NuGetHandler.ProcessRequest(HttpContext context)
   at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

Request information: 
    Request URL: http://<PROGET_SERVER>/nuget/<FEED_NAME>/Search()/$count?$filter=IsLatestVersion&searchTerm=''&targetFramework='net40|net40|net40|net40|net40|net40|net40'&includePrerelease=false 
    Request path: /nuget/Default/Search()/$count 
    Is authenticated: False 
    Authentication Type:  
    Thread account name: <USER ACCOUNT>

Thread information: 
    Thread ID: 41 
    Thread account name: <USER ACCOUNT>
    Is impersonating: False 
    Stack trace:    at System.ThrowHelper.ThrowInvalidOperationException(ExceptionResource resource)
   at Inedo.ProGet.WebApplication.SimpleHandlers.NuGet.ODataHandler.ProcessVirtualRequest(ODataResponseContext context)
   at Inedo.ProGet.WebApplication.SimpleHandlers.NuGet.ODataHandler.ProcessRequest(HttpContext context)
   at Inedo.ProGet.WebApplication.SimpleHandlers.NuGet.NuGetHandler.ProcessRequest(HttpContext context)
   at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

Product: ProGet
Version: 3.4.7

Strange, we are not able to reproduce this on any version of 3.4. Does it still happen if you try the query against a new feed with no packages in it?

Yes, I just tested by creating a new Nuget field (with all settings defaulted) and pointing the VS Nuget Package Manager to it. The logs show the same error as reported above.

How about if you go to that URL directly with a browser? I can't imagine why that would be different, but it may be worth a try.

Yeah, no difference. It just shows me an error page with the same error.

What version of SQL Server are you using?

We are using Sql Server 2008 and the database is on a different box than the Proget server.

You may want to try upgrading to 3.5.2 - it's possible this is due to an incompatibility with SQL server that we've fixed.

We upgraded to 3.5.3 but are seeing the same issue.
Over the last few days, we've tried installing different versions of Proget (from 3.3.7 onwards), as well as pointing them to a local SqlExpress instance, but nothing has worked. We get the same error everywhere.

However, we have narrowed down the issue to the $count parameter in the url.

If we run the original Search query with the $count parameter:
http://[ProGetServer]/nuget/Default/Search()/$count?$filter=IsLatestVersion&searchTerm=''&targetFramework='net40|net40|net40'&includePrerelease=false
we get the exception mentioned in the original post.

If we remove $count from that query
http://[ProGetServer]/nuget/Default/Search()?$filter=IsLatestVersion&searchTerm=''&targetFramework='net40|net40|net40'&includePrerelease=false
The query works fine and returns a list of packages.

Similarly, if we hit the server with a request for packages:
http://[ProGetServer]/nuget/Default/packages
the query runs fine and returns a list of all the packages

But if we modify it to include the $count parameter
http://[ProGetServer]/nuget/Default/packages/$count
We get the exact same exception mentioned in the first post.

So, it seems the issue is caused by a bug in the logic that returns package counts. The exception message 'Nullable object must have a value' is thrown in .net code that invokes the .Value property of a Nullable variable that is set to null.
Would it be possible for you to review that logic and see if this property is being invoked incorrectly anywhere?

Researching this issue further, we believe the cause is bad/unexpected data being returned from NuGet.org.

Can you try disabling/deleting the connector to verify? It may also be the connector filters you are using.

We've already removed the nuget.org connector, but that didn't make a difference. We have no filters or adapters configured either.

The settings for our Default package feed are:

Feed URL: /nuget/Default
Feed Disk Path: [Network share]
Bulk Import Drop Path: [Network share]
Cache Packages: NotApplicable
Active: Success
Type: NuGet

The feed is Active, and the 'Enable symbol and source server for this feed' option is enabled.

We just upgraded to v3.5.4 from v3.5.3 and the issue is now fixed.

We are still having this problem on v3.5.4. Given the following URL in a browser: http://myserver/nuget/Testing/Search()/$count?$filter=IsLatestVersion.

I tried it with a brand new feed (after removing the nuget.org connector).

It will always throw the InvalidOperationException error message:

System.InvalidOperationException: Nullable object must have a value. 
at System.Nullable`1.get_Value() 
at Inedo.ProGet.WebApplication.SimpleHandlers.NuGet.ODataHandler.ProcessVirtualRequest(ODataResponseContext context) 
at Inedo.ProGet.WebApplication.SimpleHandlers.NuGet.ODataHandler.ProcessRequest(HttpContext context) 
at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

@Garo - What happens if (after you've deleted all connectors), you go to Admin > Advanced Settings > click Save without changing anything to restart the web application? Does it work then?

No change. Still get the error message. However, the error message only shows up for non-default feeds. My "Default" feed (also no connector) will process the query just fine. When I create a new feed is when there is a problem and it errors out.

I can confirm that we are seeing the same issue as Garo with 3.5.4. The Default feed works fine, the new feeds don't. The issue is still related to the $count logic. Remove that from the url and things work fine.

Luckily, we currently only use/need the Default feed at work, so this issue is no longer blocking us.

Upon further troubleshooting, I determined that if I turn off "Symbol and Source Server" on the extra feed (with no connectors), it will work with the clean empty feed that I had created. However, I have another feed that pulls from my TeamCity v8.1 server (using the v1 api) and it generates the same error, even when I turn off the "Symbol and Source Server" and caching and clear the cache. I also tried triggering a restart of the web application by saving the Advanced Settings, but no luck. Still getting the same error.

I've found a connector caching issue that could be to blame for this. Could you try restarting the web application after deleting the connector? You can do this pretty easily by going to the Admin->All Settings page and clicking Save.

It's definitely a connector issue. If I remove the connector and reload the web app, it works. If I bring the connector back in, it fails. So any connector it seems added to a non-Default feed doesn't allow a count request.

Upgrading to ProGet 3.5.5 does not resolve the issue. It works if there is no connector associated with the non-Default feed, but once I add a connector, the query fails again.

I have confirmed that this is still a problem in ProGet 3.5.6, even after rejiggering the web application and removing / readding the connector to the feed. Any ideas on when this could be fixed?

I think we've finally been able to reproduce this. We're able to get this behavior if a connector URL is incorrect or there is a proxy issue in one of them. When testing with a connector to www.nuget.org/api/v2, it seems to work every time, but if I change it to say example.com/nothing, then I get the exact error you are describing, so I think ultimately the connector target is just returning an HTTP error code to the request, and ProGet is unfortunately failing with this as a result. I don't see why this would affect non-default feeds any differently however.

We'll have this particular issue fixed in the next release so connector HTTP errors won't result in the entire request to ProGet to fail for $count.

Unfortunately we still don't know if this is your actual issue - are you able to run something like Fiddler or Wireshark on your ProGet server to capture requests/responses to the target of your connector? If so, we may be able to confirm whether this is the problem. If not, we'll still release with this fix some time this week, so hopefully that will help you.

The URL I'm accessing is a v1 URL for TeamCity v8.1, which is: http://myservername/teamcity/guestAuth/app/nuget/v1/FeedService.svc

It populates the feed with packages correctly and I can download from the feed as well. However, when I use the magic destructo URL: http://myprogetserver:81/nuget/TeamCity/Search()/$count?$filter=IsLatestVersion. Bad things happen.

I'll try to do some wiresharking if I get a chance.

Ah, I see; please note that we do not support v1 feeds (they were already obsolete when we build ProGet in 2012) and we have no intention to. Suggest you push packages from TeamCity to ProGet instead.