@gdivis thank you!

@stevedennis

Maybe there are so many threads waiting because of this queries, but I'm not sure

@stevedennis
I set the Web.ConcurrentRequestLimit to 500, but that does not seem to help.

I found out the the memory of the ProGet AppPool grows to more than 11GB when we do about 200-300 request from one client, no other requests are involved, can I upload the memory dump somewhere?

There seems to be about 400 tasks waiting for execution (have a look at the screenshot), maybe some kind of thread pool starvation?

br

@stevedennis I will try your suggestions and let you know if I can provide more details.

Br

@stevedennis yep, I waited until the upgrade completed, it took about 10-15min, after that I started testing.

I followed the upgrade guide, any other ideas?

Br

After we have upgraded from ProGet 22.0.26 to ProGet 23.0.2 (first updating to 22.0.29 and then updating to 23.0.2 has the same effect) ProGet does no longer respond (open the WebUI of ProGet does not work, and also querying the feeds does not work, this happens as soon as we are start a few requests like these:

https://sv-int-tfs-02:8282/nuget/BuiltAsReleaseForBuildAgent/Search()?$filter=IsLatestVersion&searchTerm='Upper.ArticleConnector.Grpc.Contract'&targetFramework=''&includePrerelease=false&$skip=0&$top=3&semVerLevel=2.0.0

In the log or ProGet (WebUI) or in the Windows Event Log we got a lot of this errors:
"....... connections were in use and max pool size was reached" or "An error occurred in the web application: The operation was canceled."

Details:
URL: https://sv-int-tfs-02:8282/containers
Referrer: https://sv-int-tfs-02:8282/feeds
User: Admin
User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36
Stack trace: at System.Threading.CancellationToken.ThrowOperationCanceledException()
at Microsoft.AspNetCore.Server.IIS.Core.HttpResponseStream.ValidateState(CancellationToken cancellationToken)
at Microsoft.AspNetCore.Server.IIS.Core.HttpResponseStream.WriteAsync(ReadOnlyMemory1 source, CancellationToken cancellationToken) at Microsoft.AspNetCore.Server.IIS.Core.WrappingStream.WriteAsync(ReadOnlyMemory1 source, CancellationToken cancellationToken)
at Inedo.IO.SlimmerMemoryStream.CopyToAsync(Stream destination, Int32 bufferSize, CancellationToken cancellationToken)
at Inedo.Web.AhHttpResponse.CompleteAsync()
at Inedo.Web.AhWebMiddleware.InvokeAsync(HttpContext context)

Another error from the windows event log:
Category: Microsoft.AspNetCore.Server.IIS.Core.IISHttpServer
EventId: 2
SpanId: 560596301a29f962
TraceId: cf5b812f5bb0035c790c0cee730537b0
ParentId: 0000000000000000
RequestId: 800cb4f2-0003-f400-b63f-84710c7967bb
RequestPath: /replication

Connection ID "17582052959750763761", Request ID "800cb4f2-0003-f400-b63f-84710c7967bb": An unhandled exception was thrown by the application.

Exception:
System.OperationCanceledException: The operation was canceled.
at System.Threading.CancellationToken.ThrowOperationCanceledException()
at Microsoft.AspNetCore.Server.IIS.Core.HttpResponseStream.ValidateState(CancellationToken cancellationToken)
at Microsoft.AspNetCore.Server.IIS.Core.HttpResponseStream.WriteAsync(ReadOnlyMemory1 source, CancellationToken cancellationToken) at Microsoft.AspNetCore.Server.IIS.Core.WrappingStream.WriteAsync(ReadOnlyMemory1 source, CancellationToken cancellationToken)
at Inedo.IO.SlimmerMemoryStream.CopyToAsync(Stream destination, Int32 bufferSize, CancellationToken cancellationToken)
at Inedo.Web.AhHttpResponse.CompleteAsync()
at Inedo.Web.AhWebMiddleware.InvokeAsync(HttpContext context)
at Inedo.Web.AhWebMiddleware.InvokeAsync(HttpContext context)
at Inedo.Web.AhWebHost.<>c.<<Configure>b__17_0>d.MoveNext()
--- End of stack trace from previous location ---
at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContextOfT`1.ProcessRequestAsync()

So ProGet is completely unresponsive for minutes, often we even have to recycle the AppPool of ProGet and restart the ProGet Service.

After going back to 22.0.26 or 22.0.29 everything is working fine again.

In 22.0.0 we added the following to the shared config (C:\ProgramData\Inedo\SharedConfig\ProGet.config)

<ConnectionString>Data Source=localhost;Initial Catalog=ProGet;Integrated Security=True;Max Pool Size=1000</ConnectionString> because we had similar issues with "...connections were in use and max pool size was reached" errors, after we added Max Pool Size=1000 everything was working fine again.

I also removed Max Pool Size=1000 from the connection string in 23.0.2 but this also does not help.

Any help would highly be appreciated!

Br

@atripp thx

Hi, we are using ProGet Version 2022.1 (Build 10) and we don't get the update information when a new version is available, is there a setting that need to be changed on our side? Normally a new version is promoted very apparent?

Btw: When I go to settings --> Software Configuration --> ProGet Updates the new version is displayed.

br

@gdivis
thank you very much.

Hi, after updating to Version 2022.1 (Build 10) (coming from the last version before 2022, we are seeing this error quite often in the Diagnostic Center logs.

System.ComponentModel.Win32Exception (2): An error occurred trying to start process 'C:\Program Files\ProGet\Service~\Resources\Tools\pdbstr.exe' with working directory 'C:\Program Files\ProGet\Service'. The system cannot find the file specified.

Category: Symbol Server

Details:
System.ComponentModel.Win32Exception (2): An error occurred trying to start process 'C:\Program Files\ProGet\Service~\Resources\Tools\pdbstr.exe' with working directory 'C:\Program Files\ProGet\Service'. The system cannot find the file specified.
at System.Diagnostics.Process.StartWithCreateProcess(ProcessStartInfo startInfo)
at System.Diagnostics.Process.Start()
at Inedo.ProGet.WebApplication.SimpleHandlers.SymbolServerHandler.GetIndexedPdbAsync(Stream pdbStream, SrcSrvIni srcsrv)
at Inedo.ProGet.WebApplication.SimpleHandlers.SymbolServerHandler.ProcessRequestInternalAsync(HttpContext context)
at Inedo.ProGet.WebApplication.SimpleHandlers.SymbolServerHandler.ProcessRequestAsync(HttpContext context)

I searched for pdbstr.exe, but the only matches were in this locations:
C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\srcsrv\pdbstr.exe
C:\Program Files (x86)\Windows Kits\10\Debuggers\x86\srcsrv\pdbstr.exe

Any idea what's wrong?

br