ECONNRESET Errors when using npm-shrinkwrap via ProGet

Hello,

I am cross-posting from SO: http://stackoverflow.com/questions/34665513/econnreset-errors-when-using-npm-shrinkwrap-via-proget

Any help or suggestions would be appreciated.

We are seeing ECONNRESET issues when using a shrinkwrapped package installed through our internal ProGet server. When configuring npm registry to be the regular npm registry and shrinkwrapping a package based on that, everything installs fine. We delete node_modules and execute npm cache clean in between runs to ensure we force downloads from ProGet. Interestingly, all 1,000+ resource requests that are listed in npm-shrinkwrap.json download correctly when plugged into a file downloader.

When installing with the verbose flag, we see a number of 404s where the npm client appears to arbitrarily alter the registry url from http://<internal-url>/npm/npm to omit npm/npm for some requests, but we don't know why that occurs or whether it relates.

We use ProGet 3.8.6, npm 3.3.11 and 3.5.3 (testing on two developer machines with the same results), and node 4.2.1.

Product: ProGet
Version: 3.8.6

W're experiencing from time to time ECONNRESET while installing npm modules (with a clean before).

@Pascal, what version npm are you using and is this in conjunction with shrinkwrap or without?

Our work-around at this time is to switch to the official NPM registry prior to shrinkwrapping, so that all dependency references will bypass ProGet upon install. While that's not ideal, and while that only works since we don't depend on other custom packages we wrote (which would not be obtainable via NPM), it's not ideal as it does bypass ProGet.

Thanks Simon. npm 3.3.12 (node 5.3.0) but without shrinkwrapping. If you were able to solve your issue with the workaround you mentioned I might run into another issue.

@Pascal,

While I don't recall anything in particular about ECONNRESET issues being fixed since 3.3.12, I'd give 3.5.4 a try to see if that helps you. Fortunately, it's easy enough to switch and revert.
We dove into the npm client's code and found that errors can be a little ambiguous on the surface as npm seems to use the same error convention as node, so that ECONNRESET may be from node or maybe it's from npm. You can turn on verbose or silly logging which may give you some more info.

Good luck!

Just to add on, this is definitely something on npm's end; we've seen quite a few issues like this over the past year or so ("arbitrarily doing random things with urls that only work with npmjs.org"), that just get resolved in a future client version.

Hi Alana,

Thank you for your feedback. We had a lot of ENOENT issues back in October in which ProGet was symptomatic but npm ended up being the culprit. That issue got resolved when npm 3.3.11 was resolved (their changelog specifically addressed ENOENT errors for that version).

This time ProGet is symptomatic again, and I can't say that it's either an NPM or a ProGet issue but the URL modifications I stated - related or not - are peculiar.

Can you tell me whether Inedo has been able to reproduce this issue with shrinkwrapping and whether you can definitely rule out ProGet as a cause of this error, please?

Thank you!

We haven't tried to reproduce this as there are simply too many variables in npm/node-land to account for. However, because both ProGet and Windows competently log warnings/errors, if you aren't seeing anything there, then the issue is most certainly a npm/node problem.

It could also be a problem with your router/proxy/network/etc., but since these sorts of problems are endemic in npm (as you've noticed), I think you can be assured it's there.

Hi Alana,

In conversations with npm, their initial suspicion is that our problem lies in the npm client not throttling its requests to ProGet. For context, we issue more than 1,000 requests in our shrinkwrap file. This has not been an issue with requests to npm's registry. npm states they are considering adding throttling to the client in the future, which makes me think they may have encountered similar cases already.

Do you think it's a possibility that ProGet is simply getting overwhelmed with requests? I added a scrubbed npm silly log to issue 11125 that I opened with npm.

Thank you!

From my experience I also expect it to be an load issue on ProGet. In my use case I don't use shrinkwrap but have quite a lot of packages to install without caching, which fails on some times but not on others.

It's possible that it's a load problem, depending on how your instance is configured; but these would all be logged in at least the windows event log as a refused connection (i.e. before it gets to ProGet). This scenario could happen if your desktop has a it more bandwidth/power than the server, which is not uncommon actually.

We see load issues quite a bit on the NuGet side, as the client makes a ton of expensive requests, many of those which much be proxied/connected to NuGet.org, and those suck up all the available inbound and outbound connections.

There's not much you can change in the Integrated Web Server, but there are a lot of application pool settings you can change once you move over to IIS. Some may help, please share your results if they do.

ProGet is serving requests as fast as possible and is quite optimized/indexed/etc., but if it's a load problem, it's happening before it gets to our code. Ultimately, these clients (NuGet, npm) are effectively performing Denial of Service attacks on the server in just basic, day-to-day usage.

This is one of the many reasons organizations go with load balanced instances. That may be something you should consider, as I think it's going to be a long time before those teams learn how to make a sensible client/server protocol.

I can't imagine how many servers npmjs.org uses for handling their constant requests, but from attending talks and meeting with the team, I know NuGet.org has a massive azure farm, and is moving towards "static indexes" just to handle their DoS'ing clients...

Thanks for the response, Alex. I took a look at our IIS instance and increased the worker process count from 1 to 2 and increased the queue length from 1,000 to 1,500 but to no avail. The IIS server logs also look clean - I get 200s throughout my requests. I'll see if there are other tweaks.

Are you sure the requests are actually leaving your desktop?

Obviously The Real WTF here is doing ONE THOUSAND outbound network requests for a contiguous operation. Since they managed to write a client that does that, I think there's no chance that they can competently build a proper network request pool, let alone one that reports proper messages.

The NuGet client had a problem with this a while ago, where it was doing similar nonsense that lead to race conditions that only appeared when you used ProGet b/c the requests were being served significantly faster than NuGet.org.