Thank you.
The problem seems to be caused by Web.ConcurrentRequestLimit setting. If the server receives lots of requests and needs to query itself, these requests will be queued when the limit is reached. So they never finish in time and the original request throws the timeout exception.
This also seems to happen when the build server accesses NuGet and npm feeds on the same ProGet instance at the same time. NPM feed access also causes high CPU load. So maybe the NPM stuff causes this indirectly....?
I get some "The SSL connection could not be established" errors with "Authentication failed because the remote party sent a TLS alert: 'DecryptError'" in inner exception details when ProGet tries to access an upstream NuGet feed using the NuGet connector. The URL is
https://nuget.devexpress.com/<secret-removed>/api/FindPackagesById()?id=%27Newtonsoft.Json%27&semVerLevel=2.0.0
Callstack:
Request URL: https://nuget.devexpress.com/<secret-removed>/api/FindPackagesById()?id=%27Newtonsoft.Json%27&semVerLevel=2.0.0
System.Net.WebException: The SSL connection could not be established, see inner exception.
---> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
---> System.Security.Authentication.AuthenticationException: Authentication failed because the remote party sent a TLS alert: 'DecryptError'.
---> System.ComponentModel.Win32Exception (0x80090326): The message received was unexpected or badly formatted.
--- End of inner exception stack trace ---
at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(HttpRequestMessage request)
at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.AuthenticationHelper.SendWithAuthAsync(HttpRequestMessage request, Uri authUri, Boolean async, ICredentials credentials, Boolean preAuthenticate, Boolean isProxyAuth, Boolean doRequestAuth, HttpConnectionPool pool, CancellationToken cancellationToken)
at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
at System.Net.HttpWebRequest.SendRequest(Boolean async)
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location ---
at Inedo.ProGet.Feeds.Connector.GetDirectResponseAsync(String url)
at Inedo.ProGet.Feeds.Connector.GetCachedResponseAsync(String url)
at Inedo.ProGet.Feeds.Connector.GetCachedTextReaderAsync(String url, Boolean bufferEntireResponse)
at Inedo.ProGet.Feeds.NuGet.V2.NuGetFeedClient.PerformRequestWithAbsoluteUrlAsync(String fullUrl)
Latest version of ProGet, latest Windows Server 2022 with all updates installed. No proxy is used. If I open the URL from a local browser, connection works fine.
The most interesting thing is that the connector seems to work. If I list packages from the feed that uses this connector, I get valid results and no errors in log. But if I run the "Connector Health Check", I get "Connector devexpress.com error: Input string was not in a correct format" error....
I have a NuGet "default" feed that is connected to the standard nuget.org feed and also includes connectors to three "local" NuGet feeds. I get lots of "The operation has timed out." errors when the build server tries to restore packages:
The callstack is always the same:
Request URL: https://nuget.local/nuget/internal/FindPackagesById()?id=%27Microsoft.Win32.SystemEvents%27&semVerLevel=2.0.0
System.Net.WebException: The operation has timed out.
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location ---
at Inedo.ProGet.Feeds.Connector.GetDirectResponseAsync(String url)
at Inedo.ProGet.Feeds.Connector.GetCachedResponseAsync(String url)
at Inedo.ProGet.Feeds.Connector.GetCachedTextReaderAsync(String url, Boolean bufferEntireResponse)
at Inedo.ProGet.Feeds.NuGet.V2.NuGetFeedClient.PerformRequestWithAbsoluteUrlAsync(String fullUrl)
Looks like there are some "deadlock" issues when ProGet tries to read from feeds that are also handled by the same ProGet instance?