Welcome to the Inedo Forums! Check out the Forums Guide for help getting started.
If you are experiencing any issues with the forum software, please visit the Contact Form on our website and let us know!
ProGet 2023 - IIS App pool stopping
-
After upgrading to ProGet 2023 we are seeing errors which causes the IIS app pool to stop, and it will start back up after someone tries to download a package or us the UI, but takes 90 sec - 2 minute to start back up. This is what I am seeing in the event viewer which coincides with with the app pool stops (we are on windows server 2022):
Category: Microsoft.AspNetCore.Server.IIS.Core.IISHttpServer
EventId: 2
SpanId: 6f3ab11a2a3bf2ce
TraceId: ea531c1240218f307196736526edf8bb
ParentId: 0000000000000000
RequestId: 40001e29-0001-d800-b63f-84710c7967bb
RequestPath: /Connection ID "15564440317292715559", Request ID "40001e29-0001-d800-b63f-84710c7967bb": An unhandled exception was thrown by the application.
Exception:
System.OperationCanceledException: The operation was canceled.
at System.Threading.CancellationToken.ThrowOperationCanceledException()
at Microsoft.AspNetCore.Server.IIS.Core.HttpResponseStream.ValidateState(CancellationToken cancellationToken)
at Microsoft.AspNetCore.Server.IIS.Core.HttpResponseStream.WriteAsync(ReadOnlyMemory1 source, CancellationToken cancellationToken) at Microsoft.AspNetCore.Server.IIS.Core.WrappingStream.WriteAsync(ReadOnlyMemory1 source, CancellationToken cancellationToken)
at Inedo.IO.SlimmerMemoryStream.CopyToAsync(Stream destination, Int32 bufferSize, CancellationToken cancellationToken)
at Inedo.Web.AhHttpResponse.CompleteAsync()
at Inedo.Web.AhWebMiddleware.InvokeAsync(HttpContext context)
at Inedo.Web.AhWebMiddleware.InvokeAsync(HttpContext context)
at Inedo.Web.AhWebHost.<>c.<<Configure>b__18_0>d.MoveNext()
--- End of stack trace from previous location ---
at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContextOfT`1.ProcessRequestAsync()
-
Hello,
I am copy-pasting the same reply made to EDO-10211 -- feel free to reply to one or the other.
We'll do our best to help, but based on the information you provided, it sounds like your server might be "overloaded" and have more traffic than it can handle. I would recommend trying out ProGet 2023.31 (releasing later today) which has a few more performance tweaks, but if that doesn't solve the issue than it will involve looking at ways to expand server/clsuter capacity or reduce/throttle traffic.
The error you shared is unfortunately unrelated, and is just a generic "client disconnected" error - it will occur if a a client (web browser, etc) disconnects prior a request being received. These should not be logged, but due to a known issue in .NET6 they are. This should be fixed in .NET8 (ProGet 2024).
Best place to check is ProGet Diagnostic Center, under Admin. That will have the most relavent errors.
Best,
Alana
-
@atripp Thanks for the reply. After more data gathering I think only once the I found the app pool stopped so it seems the unresponsiveness is something else. I thought something might be forcing a recycle because it is roughly the same amount of time. I cannot confirm at this time that it is being stopped or recycling.
The .NET exception may be unrelated. but it started post-upgrade. And the unresponsiveness was noted again today (Saturday) when no one is online working. I navigated to UI, waited two minutes for it respond, and then sure enough a new Microsoft.AspNetCore.Server.IIS.Core.IISHttpServer exception is in the event viewer. So if not the root cause of anything, the timing sure makes it seem at least as another effect.
There are no related messages in diagnotics center. I wondering if there is a way to do some tracing.
-
Hi @rick-kramer_9238 ,
It's possible that the IIS App-pool is automatically shutting down, which can lead to a "warm up" time that's required on a first request. However, this is typically measured in seconds in slow cases, not minutes. So it shouldn't take that long to "warm up".
Unfortunately this isn't easy to troubleshoot, and it seems to be related to some kind of strange IIS configuration. We've heard of users solving it lots of different ways, from switching servers to uninstalling IIS, to switching to the Integrated Web Server. But no idea what actually works.
The easiest thing to do, you may want to consider switching to the Integrated Web Server. Microsoft recommends that over IIS these days as well. The quickest way to do this is to uninstall ProGet (this does not delete database or packages), then reinstall to use the integrated web server. Make sure to point to same database at install time. You can configure HTTPS later.
Best,
Alana
-
@atripp Thanks, I will continue to monitor. We did not get the .NET error this morning. There was no server reboot but there was an automated IIS reset. We are going to also set up a synthetic monitor in Dynatrcae on the UI.
-
@atripp Do you have any thoughts on the changing the Start Mode and the idle timeout on the app pool? Or Preload Enabled on the Site?
-
Hi @rick-kramer_9238 , you could certainly try to enable those and see if they help or make a change.
-
@atripp After doing more testing, I do not think this has anything to do with IIS. I think the login process is what is spinning for 3-4 minutes on initial testing in the morning. We have Windows Integrated Auth enabled wich is what we had been using before upgrade. It looks like Username/Password is also enabled now, not sure if that was enabled before the upgrade. I would be interested in any thoughts you might have on this. thanks
-
Hi @rick-kramer_9238 ,
We have seen cases where LDAP/AD is painfully slow; one common reason is that "recursive group search" is enabled.
But before digging into that, I would just temporary disable Windows Auth (do this in IIS) and disable LDAP/AD, so that you're only loggin in with a built-in username/password. If it's still really slow, then you know it's unrelated.
Best,
Alana
-
@atripp If we switch to using integrated web server I would probably upgrade from 2023.30 to 24.0.1. If we would happen to need to rollback, would a database restore be needed-- are there database changes moving between these version? Thanks!
-
Hi @rick-kramer_9238 ,
I wouldn't recommend upgrading to ProGet 2024 solely for that reason, as it's a major release and if there were issues, it'd be easier to isolate Integrated Web Server vs Regression.
Here are the upgrade notes:
https://docs.inedo.com/docs/proget-upgrade-2024That said, if you need to rollback to ProGet 2023, you can do so without restoring the database by simply using the Inedo Hub. While there are database schema changes, they are all backwards-compatible with ProGet 2023, which means you can safely rollback your ProGet installation if there's a showstopper bug, and then upgrade later.
However, you should backup your database as an extra precaution anyway.
Thank you,
Steve
-
@atripp After re-installed with the IWS, we want to set up HTTPS and bind the the existing cert for the DNS entry we use for our ProGet instance. Is it necessary to do all the steps on https://docs.inedo.com/docs/installation-windows-https-support, or can we just register the certificate underneat "HTTPS Binding to a Hostname"
-
In all cases, you'll need to configure ProGet to listen to a certain URL; this basically involves editing the ProGet configuration file, which the web page will do for you (or you can edit yourself).
If you're going to do port sharing (i.e. have multiple applications on the server listen/respond
on port 443 for example), the you also need register the URL and certificate with Windows using the netssh commands.If you're not doing port sharing (i.e. ProGet is only application listening on 443) then no need to use netsh.
Hope that helps,
Alana
-
@atripp Thanks, we got the cert added and https working fine, but after moving away from IIS and running ProGet self-hosted, if anyting the logging time is worse. Everytime I bring up the UI it re-auths me and spins for minutes before opening. There are no messages in the Diagnostic Center. Is there any place I can turn on additional logging to help diagnose this?
-
The issue sounds like it's related to LDAP configuration (i.e. slow queries to your LDAP/AD server), but it's hard to say. This wouldn't behave any differently in IIS for IWS.
The first thing I would try is disabling LDAP / AD. And Windows Integrated Authentication (if you have it enabled). If the server is still slow, then I would try http:// instead of https://
Once you've identified where the slowness is coming from, we can address it. The most common issue with LDAP is recursive/nested group searches - especially when there are like thousands of groups and everyone's a member of something.
I would "play" with your LDAP settings and try to isolate why it's so slow.
Thanks,
Steve
