Service Unavailable when downloading connector packages through nuget.exe

When attempting to restore a package using nuget.exe/Xamarin Studio/Visual Studio 2017, we are seeing the error "ServiceUnavailable".

Entering the URL directly into a browser shows that we are getting a 503 "The service is unavailable". An example URL would be: https://<progethostname>/nuget/NuGet.org/package/Scandit.BarcodePicker.Xamarin/5.5.1.10

We have a manually configured, authenticated proxy that seems to be working (i.e. it tests correctly and we can browse the feed - and even Pull to ProGet successfully), but for some reason we can't restore packages at the moment. We are also seeing issues with the npmjs.org connector which always fails to update its cache.

Is there anyway to diagnose the issue further? There doesn't seem to be any exception logs available for us to view.

Product: ProGet
Version: 4.8.9

ProGet does not issue such a message; IIS does, and can happen in a few cases, such as when an application pool is stopped but you visit a website that requires it.

However, it sounds like you can visit other URLS in ProGet, so I suspect something else... like your proxy... is not actually working the way it should be.

Hi Alana, I'm sure it's something to do with the proxy, but there doesn't seem to be a way to diagnose what it is.

ProGet is simply returning 503 Service Unavailable and there are no logs to determine what URL it is failing to connect to, or why it is giving up on the connection.

We really need clearer error logs for these sorts of issues.

As I mentioned, ProGet does not issue 503 error messages. It's actually your proxy server, or something else before ProGet. I don't know your network configuration, but in the case of a proxy, the proper working flow is like this:

Request: Web browser >>>>>>> Proxy Server >>>>>> ProGet

Response: Web browser <<<<<< Proxy Server <<<< ProGet

Your web browser does not "know" it's talking to a Proxy Server, and ProGet does not "know" it's talking to a Proxy Server.

But in your case, your proxy server is misconfigured, so the flow is only like this for some requests:

Request: Web browser >>>>>>> Proxy Server

Response: Web browser <<<<<< Proxy Server (error)

Basically, ProGet is not receiving any information, and so it can't send or log anything.

You really need to solve this on the Proxy side.

Hi Alana,

I don't think this is correct.

My web browser is connecting directly to ProGet - using the URL https://<progetserver>/nuget/<feedname>/package/Scandit.BarcodePicker.Xamarin/5.5.1.10 - and is getting a direct 503 back from that connection to ProGet.

It is possible - but I have no way of knowing - that ProGet is doing some form of proxying and sending the proxy response directly to the browser, but the browser connection is definitely to the ProGet server. Because of this, I would expect ProGet to return a useful error message rather than an opaque "The service is unavailable", which tells me nothing.

In your example, the flow is actually:

Request: Web browser >>>>> ProGet >>>>> Proxy? >>>>> Source
Response: Web browser <<<<< ProGet (The service is unavailable) <<<<< ?????

A proxy server acts as a firewall, which means it intercepts requests/responses to and clients. Like a firewall, there is no way for a server on the "other side" of the proxy (such as ProGet) to bypass that, and go directly to your browser.

Basically, your browser connects to the proxy server on your network (this happens transparently, so even if you type in the url, your computer is instead going to the proxy). The proxy server connects then to the intended site (ProGet). The intended site then returns the response to Proxy, which then returns it to your browser.

ProGet does not perform "some kind of proxying" or anything like that. Moreover, the ProGet software also does not, under any circumstance whatsoever, return a "503" status at all. So, that "rather opaque" error message is coming elsewhere (not ProGet).

It's like, if you disconnect from wifi, and you get "cannot resolve host google.com" in your browser; that message doesn't come from google.com (even though it's in your address bar), but your browser.

Hi Alana,

My browser is absolutely, 100% not using a proxy to connect to ProGet.

ProGet itself is configured to use a proxy to connect to the outside world.

There is an error somewhere between ProGet, the proxy it uses, NuGet and wherever the package I am requesting is hosted. Instead of getting a sensible error message from ProGet, I get "503 Service Unavailable". This HTTP error is delivered to my browser through a connection that is direct to ProGet (proven by a Wireshark trace).

It appears that when you hit a package link on ProGet, it sets up a connection to the package host and then streams the result of that call directly to the browser. Ideally, it would only do this for a http 200, and not for an error state.

The problem is that I have no idea what is actually throwing the error, because it's not logged anywhere. In fact, from your description, it sounds like ProGet isn't even aware that the connection it is forwarding has even errored in the first place. To me, this is a ProGet bug because the detail of ProGet retrieving a package from a package source should not be important to the end user.

Oh I see; I misunderstood when you said "we have a manually configured, authenticated proxy" I thought you had meant your browser would be configured to use this too.

In any case, as I mentioned before, there is no possible way for ProGet to ever return a 503 status. So if you see 503, then the request is never making it to the ProGet software.

It's quite common for connectors to fail (internet outage, NuGet.org outage, etc), so this scenario is handled. If a connector returns a 503, then ProGet will log this error, and may send the browser a different error message (it really depends on the context). But in no case, at all, will ProGet send "503 Service Unavailable".

As I mentioned at first, IIS will return this message if the application pool has stopped. In this case, the request can't be sent to ProGet, because IIS isn't running ProGet at the time. I would check WIndows Error logs to see why IIS is stopping application pools.

Alana,

Can you please explain this entry in my IIS logs for the ProGet website, then?

2017-12-14 09:14:27 <ipaddress> GET /nuget/NuGet.org/package/Scandit.BarcodePicker.Xamarin/5.5.1.10 - 443 kirsty <ipaddress> <useragent> 503 0 0 1575

That is clearly ProGet returning a 503 to the browser in response to a request for a package hosted by NuGet.org. Note that the AppPool has not crashed and the rest of the site is fully operational.

I am happy to help the development team track down the problem, but I need you to acknowledge that I am not making it up first.

As I mentioned, ProGet does not, nor cannot return a 503 message.

What's happening here is that IIS receiving the request, immediately responding with a 503, then logging the request. The request is never being forwarded to ProGet. This is why you can't find any information in ProGet's logs about it.

Here is a way you can debug 503 messages in IIS:
http://mvolo.com/where-did-my-iis7-server-go-troubleshooting-503-quotservice-unavailablequot-errors/. Based on you saying the app pool is running, I think it's some other configuration that's causing it.. but I can't know what it is.

Of course, if you search "503 errors in IIS" you can find more methods of debugging it.

Once you identify the underlying reasons, it will hopefully be obvious what's causing it. If not please post the error messages or info you find, and I can try to help.

Hi Alana,

I really don't think 1575ms is "immediately". ProGet is doing "something" for 1.5 seconds before it gives up and returns a 503 to the browser.

Look, I'm going to skip the next 50 messages going backwards and forwards because I have managed to debug it myself, but you really need to listen and send this information to the developers:

The test

The package Scandit.BarcodePicker.Xamarin is listed on NuGet.org's main feed. However, the .nupkg is actually hosted at https://az320820.vo.msecnd.net/packages/scandit.barcodepicker.xamarin.5.5.1.10.nupkg

We will be downloading the package through a nuget feed hosted in ProGet with a standard connector pointing to nuget.org. The test url is https://<progetserver>/nuget/<feedname>/package/Scandit.BarcodePicker.Xamarin/5.5.1.10

Scenario 1 - ProGet is configured to use a proxy that can connect to nuget.org but not msecnd.net

In this scenario, ProGet spends 1500ms trying to connect to msecnd.net, times out and returns 503 Unavailable to the calling browser.

Scenario 2 - ProGet is configured to use a proxy that allows connections to both nuget.org and msecnd.net

Everything works correctly!

Conclusion

This should be easy to reproduce at your end. It would be helpful if instead of a 503 we could actually get a sensible error message saying why the package can't be retrieved.

I am one of the developers, which is why I can assure to you that ProGet is not issuing a 503. This is IIS.

However, based on the new information you've provided, it's likely happening because of an abrupt Application Pool termination such as a access violation, stackoverflow, etc. In such a situation, it's not possible for ProGet to log anything; the program (AppPool) immediately crashes. Then IIS issues the 503 error.

However, IIS should be logging these in the Windows Event log; so if you can look into what it is, then we can try to debug it further.

This is an easy scenario for us to set-up, and it's a common error condition, but we cannot reproduce it.