1. Home
  2. Categories
  3. Support
  4. Error: HttpException Server Too Busy

Welcome to the Inedo Forums! Check out the Forums Guide for help getting started.

If you are experiencing any issues with the forum software, please visit the Contact Form on our website and let us know!

Error: HttpException Server Too Busy

  • G

    Hello,

    I recently took over an old ProGet server (v. 4.8.8).
    After numerous failed attempts at upgrading it, we decided to stick with the current version until we can upgrade it successfully.

    Now, the issue I'm currently having is that the web service stops working every 2 hours or so.
    I get the below warning in event viewer for pretty much every request when it stops working, but I really don't see what would cause it to be "too busy".
    The new server it is hosted on has twice as much resources as the old one and runs on Windows Server 2019, so there should be no hardware limitations causing this.

    Any ideas on what could be causing this behaviour?

    Error Example:

    The description for Event ID 1309 from source ASP.NET 4.0.30319.0 cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event: 

3005
An unhandled exception has occurred.
2022-02-08 12:16:33
2022-02-08 11:16:33
0c0b1c92e30d4211bf9d8008ad971841
29008
29007
0
e906d479-1-132887818818008266
Full
/
C:\Program Files\ProGet\WebApp\
P-NUGET-OBJR
5520
ProGet.Service.exe
NT AUTHORITY\NETWORK SERVICE
HttpException
Server Too Busy
   at System.Web.HttpRuntime.RejectRequestInternal(HttpWorkerRequest wr, Boolean silent)


http://nuget.****.****/npm/****-npms/****-****/-/****-****-4.3.0.tgz
/npm/****-npms/****-****/-/****-****-4.3.0.tgz
10.****.****.****
False
NT AUTHORITY\NETWORK SERVICE
163
NT AUTHORITY\NETWORK SERVICE
False
   at System.Web.HttpRuntime.RejectRequestInternal(HttpWorkerRequest wr, Boolean silent)
    0 atripp 1 Reply
  • G

    ALSO:

    The few seconds it is working, the following error is returned in the hundreds (for different packages, but same gist):

    An error occurred processing a GET request to http://nuget.****.TLD/npm/****-npms/***.js/-/***.js-4.12.0.tgz: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
 Details: System.InvalidOperationException: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
at System.Data.Common.ADP.ExceptionWithStackTrace(Exception e)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Inedo.Data.SqlServerDatabaseContext.<CreateConnectionAsync>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Inedo.LazyAsync`1.<GetValueAsync>d__12.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Inedo.Data.DatabaseContext.<ExecuteInternalAsync>d__37.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Inedo.ProGet.Data.DB.Context.<NpmPackages_GetPackageAsync>d__313.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Inedo.ProGet.Feeds.Npm.NpmFeed.<TryGetPackageAsync>d__16.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Inedo.ProGet.WebApplication.FeedEndpoints.Npm.NpmPackageDownloadHandler.<TryProcessRequestAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Inedo.ProGet.WebApplication.FeedEndpoints.Npm.NpmHandler.<ProcessRequestAsync>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Inedo.ProGet.WebApplication.FeedEndpoints.FeedEndpointHandler.FeedRequestHandler.<ProcessRequestAsync>d__8.MoveNext()
    0
  • atripp
    inedo-engineer

    Hi @galaxyunfold ,

    Based on the symptoms your describing, it sounds like the problem is load-related. How many developers/machines are using this instance of ProGet?

    When you have a handful of engineers doing package restores with tools like npm, it's similar to a "DDoS" on the server -- the npm client tool makes hundreds of simultaneous requests to the server. And the server then has to make database connections, and often connections out to npmjs.org, etc. The network queues get overloaded, and then you get symptoms like this.

    Ultimately load-related issues are from a lack of network resources, not CPU/ram. You can reduce connections (throttle end-users, remove connectors, etc.), but the best bet is going with a high-availability / load-balanced configuration.

    I would also recommend upgrading, as there's been a lot of performance improvements in the 4-5 years since ProGet v4 was released.

    Alana

    0 G 1 Reply
  • G

    @atripp Thank you very much for your reply.

    It seems the issue was actually related to Windows Long Paths and had nothing to do with load at all.
    While this is an older version of ProGet (currently we're up to v5.2.0), it is surprising that the error logs would give such errors for something that had nothing to do with it.

    None the less, with the exception of issues with the legacy connectors and/or quirk feeds, all seems to be running OK now.

    Are there any specifications for what is required based on how many users are using NPM (and NuGet etc.)?
    Load-balancing for a few users seems rather overkill to me, can ProGet not handle a handful of users without a proxy in between?

    0 atripp 1 Reply
  • atripp
    inedo-engineer

    Hi @galaxyunfold ,

    The "Timeout expired" errors are indeed a result of database or network connectivity issues. It's possible to create connector loops (A ->B -> C -> A) that will yield this behavior as well.

    The "server too busy" is an internal IIS error, and it can be much more complicated. It's rarely related to load, and is more related to performing an operation during an application pool recycle. Frequently crashing application pools will see this error frequently.

    There are a lot of factors that determine load, and how you configure ProGet (especially with connectors and metadata caching) makes a big difference. But in general, it starts to make sense at around 50 engineers. At 250+ engineers, it makes sense not to go load-balanced / high-availability.

    Here is some more information: https://blog.inedo.com/proget-free-to-proget-enterprise

    Cheers,
    Alana

    0
Log in to reply
 
Inedo Website Home • Support Home • Code of Conduct • Forums Guide • Documentation