@rhessinger Ok, then I understand. We will look into our options then. Depending on Mono fixing this seems like a bad solution as it seems to have appeared for quite some time.
Thanks for all your help!
What is the timeline of your Mono to .NET 5 conversion?
Hi @rhessinger
Ok, the issue you found sounds interesting, but it seems to be more connected to the other problem we have: https://forums.inedo.com/topic/2887/proget-in-docker-linux-hanging-after-using-all-memory/6
And we are currently already restarting the docker container once a day to make sure that we don't run out of memory.
And looking into the memory graphs it looks like we experience the connection issue even if the memory consumption looks ok (Green bars = scheduled restarts, red bars = connector problems, yellow graph = available memory, green graph = free memory): 
-- Magnus
Now it misbehaves...
Active connections: 6
server accepts handled requests
115659 115659 179524
Reading: 0 Writing: 1 Waiting: 5
Accessing feeds/default will show a "There was an error with a connector: The operation has timed out." All other feeds work as expected.
ubuntu@proget:~$ sudo ps huH p 18819 |wc -l
33
In the admin interface:
Request URL: https://nuget.org/api/v2/Search()?$top=25&$filter=IsAbsoluteLatestVersion&searchTerm=%27%27&targetFramework=%27%27&includePrerelease=true&semVerLevel=2.0.0
System.Net.WebException: The operation has timed out.
at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task`1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func`1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.HttpWebRequest.EndGetResponse (System.IAsyncResult asyncResult) [0x00020] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Threading.Tasks.TaskFactory`1[TResult].FromAsyncCoreLogic (System.IAsyncResult iar, System.Func`2[T,TResult] endFunction, System.Action`1[T] endAction, System.Threading.Tasks.Task`1[TResult] promise, System.Boolean requiresSynchronization) [0x0000f] in <c453bf849bb84e719dbec0475e99db95>:0
--- End of stack trace from previous location where exception was thrown ---
at Inedo.ProGet.Feeds.StrongConnector.GetDirectResponseAsync (System.String url) [0x000eb] in <dacb7de90ce24964b2da21b67ac5e1d1>:0
at Inedo.ProGet.Feeds.StrongConnector.GetCachedResponseAsync (System.String url) [0x000a8] in <dacb7de90ce24964b2da21b67ac5e1d1>:0
at Inedo.ProGet.Feeds.StrongConnector.GetCachedTextReaderAsync (System.String url) [0x00075] in <dacb7de90ce24964b2da21b67ac5e1d1>:0
at Inedo.ProGet.Feeds.NuGet.NuGetFeedClient.PerformRequestWithAbsoluteUrlAsync (System.String fullUrl) [0x000d9] in <dacb7de90ce24964b2da21b67ac5e1d1>:0
strace doesn't show anything obvious.
Restart docker container and now the /feed/default shows no problem. nginx status:
Active connections: 6
server accepts handled requests
115829 115829 180229
Reading: 0 Writing: 1 Waiting: 5
@rhessinger I will do that.
One note, it seems that running on debug logging level makes it a little more unstable. We are back to normal "warning" now and now it seems to be back at our regular "once a day" frequency. With debug it felt like 5 times a day.
@rhessinger Hi, if I run it on the host (or in the actual container) it seems to vary between 25-45, haven't seem anything different between when it behaves bad:
ubuntu@proget:/opt/provision/additional$ sudo ps huH p 12436 | wc -l
29
ubuntu@proget:/opt/provision/additional$ sudo ps huH p 12436 | wc -l
44
Performing strace on the mono process gives a lot of:
[pid 26706] recvfrom(39, "", 8, 0, NULL, NULL) = 0
[pid 26706] recvfrom(39, "", 8, 0, NULL, NULL) = 0
[pid 26706] recvfrom(39, "", 8, 0, NULL, NULL) = 0
[pid 26706] recvfrom(39, "", 8, 0, NULL, NULL) = 0
[pid 26706] recvfrom(39, "", 8, 0, NULL, NULL) = 0
[pid 26706] recvfrom(39, "", 8, 0, NULL, NULL) = 0
[pid 26706] recvfrom(39, "", 8, 0, NULL, NULL) = 0
Then I found this: https://stackoverflow.com/questions/24000534/tracing-socket-file-descriptor-back-to-bound-address
So the next step will be to see if I can find out which socket that is actually appearing in the strace.
One interesting observation is that when the system "freezes" there seems to be no traffic at all (12:30):
Now it's seems flaky again:
nginx status:
Active connections: 57
server accepts handled requests
462208 462208 840656
Reading: 0 Writing: 30 Waiting: 27
Main web UI can't be reached at all.
Ran sudo kill -QUIT $MONO_PID and got myself a huge thread dump with a lot of entries like this (let me know if you want me to supply the entire dump, (it's more then the allowed 32k chars):
"Thread Pool Worker" at <unknown> <0xffffffff>
at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0
@rhessinger 99% NuGet connector errors.
@rhessinger said in Connector errors, timeout, restart needed.:
Also, how many connectors do you have on your NuGet feed?
Sorry, I missed this question... We have four connectors (but only uses three of them): 
And when I looked that up I noticed that the one we don't use was marked with error, removed it.
@rhessinger We have enabled the stub_status for Nginx, when the problem appeared just now the status showed the following page:
Active connections: 15
server accepts handled requests
428494 428494 756561
Reading: 0 Writing: 2 Waiting: 13
We have some network connection monitoring with prometheus for the host and they show nothing specific (see attached image)
@rhessinger I will look into that!
Meanwhile, I tested to run "kill -QUIT $MONO_PID" to get some thread dumps from the mono process when the problem appears.
"Thread Pool Worker" at <unknown> <0xffffffff>
at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0
"Thread Pool Worker" at <unknown> <0xffffffff>
at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0
"Thread Pool Worker"
"Thread Pool Worker" at <unknown> <0xffffffff>
at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0
"Thread Pool Worker" at <unknown> <0xffffffff>
at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0
"Thread Pool Worker" at <unknown> <0xffffffff>
at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0
"Thread Pool Worker" at <unknown> <0xffffffff>
at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0
"Thread Pool I/O Selector" at <unknown> <0xffffffff>
at (wrapper managed-to-native) System.Threading.ThreadPool.RequestWorkerThread () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.ThreadPoolWorkQueue.EnsureThreadRequested () [0x0001f] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.ThreadPoolWorkQueue.Enqueue (System.Threading.IThreadPoolWorkItem,bool) [0x00071] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.ThreadPool.UnsafeQueueCustomWorkItem (System.Threading.IThreadPoolWorkItem,bool) [0x00007] in <c453bf849bb84e719dbec0475e99db95>:0
at (wrapper runtime-invoke) <Module>.runtime_invoke_void_object_byte (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0
"Thread Pool Worker" at <unknown> <0xffffffff>
at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0
"Thread Pool Worker" at <unknown> <0xffffffff>
at (wrapper managed-to-native) System.Net.Sockets.Socket.Receive_internal (intptr,byte*,int,System.Net.Sockets.SocketFlags,int&,bool) [0x00000] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.Sockets.Socket.Receive_internal (System.Net.Sockets.SafeSocketHandle,byte*,int,System.Net.Sockets.SocketFlags,int&,bool) [0x0000c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.Sockets.Socket.Receive (byte[],int,int,System.Net.Sockets.SocketFlags,System.Net.Sockets.SocketError&) [0x0002e] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.Sockets.Socket.Receive (byte[],int,int,System.Net.Sockets.SocketFlags) [0x00000] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.Sockets.NetworkStream.Read (byte[],int,int) [0x00065] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Data.SqlClient.SNI.SslOverTdsStream/<ReadInternal>d__11.MoveNext () [0x000e8] in <0864334e7e474248b37afca9b637daa9>:0
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1<int>.Start<System.Data.SqlClient.SNI.SslOverTdsStream/<ReadInternal>d__11> (System.Data.SqlClient.SNI.SslOverTdsStream/<ReadInternal>d__11&) [0x0002c] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Data.SqlClient.SNI.SslOverTdsStream.ReadInternal (byte[],int,int,System.Threading.CancellationToken,bool) [0x0004d] in <0864334e7e474248b37afca9b637daa9>:0
at System.Data.SqlClient.SNI.SslOverTdsStream.Read (byte[],int,int) [0x00009] in <0864334e7e474248b37afca9b637daa9>:0
at Mono.Net.Security.MobileAuthenticatedStream/<>c__DisplayClass66_0.<InnerRead>b__0 () [0x0002b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Threading.Tasks.Task`1<int>.InnerInvoke () [0x0000f] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.Tasks.Task.Execute () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.Tasks.Task.ExecutionContextCallback (object) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00071] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.Tasks.Task.ExecuteWithThreadLocal (System.Threading.Tasks.Task&) [0x00034] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.Tasks.Task.ExecuteEntry (bool) [0x0004a] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.Tasks.Task.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0
"Thread Pool Worker" at <unknown> <0xffffffff>
at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0
"Timer-Scheduler" at <unknown> <0xffffffff>
at (wrapper managed-to-native) System.Threading.WaitHandle.Wait_internal (intptr*,int,bool,int) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.WaitHandle.WaitOneNative (System.Runtime.InteropServices.SafeHandle,uint,bool,bool) [0x00044] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.WaitHandle.InternalWaitOne (System.Runtime.InteropServices.SafeHandle,long,bool,bool) [0x00014] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.WaitHandle.WaitOne (long,bool) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.WaitHandle.WaitOne (int,bool) [0x00019] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.WaitHandle.WaitOne (int) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.Timer/Scheduler.SchedulerThread () [0x0003c] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.ThreadHelper.ThreadStart_Context (object) [0x00014] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00071] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object) [0x0002b] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.ThreadHelper.ThreadStart () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
at (wrapper runtime-invoke) object.runtime_invoke_void__this__ (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0
"Thread Pool Worker" at <unknown> <0xffffffff>
at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0
"<unnamed thread>" at <unknown> <0xffffffff>
at (wrapper managed-to-native) Interop/Sys.Read (intptr,byte*,int) [0x00000] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at Interop/Sys.Read (Microsoft.Win32.SafeHandles.SafeFileHandle,byte*,int) [0x00012] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.IO.CoreFX.FileSystemWatcher/RunningInstance.TryReadEvent (System.IO.CoreFX.FileSystemWatcher/RunningInstance/NotifyEvent&) [0x00027] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.IO.CoreFX.FileSystemWatcher/RunningInstance.ProcessEvents () [0x002dc] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.IO.CoreFX.FileSystemWatcher/RunningInstance/<>c.<Start>b__14_0 (object) [0x00000] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Threading.Tasks.Task.InnerInvoke () [0x00025] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.Tasks.Task.Execute () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.Tasks.Task.ExecutionContextCallback (object) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00071] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.Tasks.Task.ExecuteWithThreadLocal (System.Threading.Tasks.Task&) [0x00034] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.Tasks.Task.ExecuteEntry (bool) [0x0004a] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.Tasks.ThreadPoolTaskScheduler/<>c.<.cctor>b__10_0 (object) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.ThreadHelper.ThreadStart_Context (object) [0x00025] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00071] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object) [0x0002b] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.ThreadHelper.ThreadStart (object) [0x0000f] in <c453bf849bb84e719dbec0475e99db95>:0
at (wrapper runtime-invoke) <Module>.runtime_invoke_void__this___object (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0
"Thread Pool Worker" at <unknown> <0xffffffff>
at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0
"Thread Pool Worker" at <unknown> <0xffffffff>
at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0
"Thread Pool Worker" at <unknown> <0xffffffff>
at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0
"Thread Pool Worker" at <unknown> <0xffffffff>
at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0
"Thread Pool Worker" at <unknown> <0xffffffff>
at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0
"Thread Pool Worker" at <unknown> <0xffffffff>
at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0
"Thread Pool Worker" at <unknown> <0xffffffff>
at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0
"Finalizer"
"<unnamed thread>" at <unknown> <0xffffffff>
at (wrapper managed-to-native) Interop/Sys.Read (intptr,byte*,int) [0x00000] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at Interop/Sys.Read (Microsoft.Win32.SafeHandles.SafeFileHandle,byte*,int) [0x00012] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.IO.CoreFX.FileSystemWatcher/RunningInstance.TryReadEvent (System.IO.CoreFX.FileSystemWatcher/RunningInstance/NotifyEvent&) [0x00027] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.IO.CoreFX.FileSystemWatcher/RunningInstance.ProcessEvents () [0x002dc] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.IO.CoreFX.FileSystemWatcher/RunningInstance/<>c.<Start>b__14_0 (object) [0x00000] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Threading.Tasks.Task.InnerInvoke () [0x00025] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.Tasks.Task.Execute () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.Tasks.Task.ExecutionContextCallback (object) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00071] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.Tasks.Task.ExecuteWithThreadLocal (System.Threading.Tasks.Task&) [0x00034] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.Tasks.Task.ExecuteEntry (bool) [0x0004a] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.Tasks.ThreadPoolTaskScheduler/<>c.<.cctor>b__10_0 (object) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.ThreadHelper.ThreadStart_Context (object) [0x00025] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00071] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object) [0x0002b] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.ThreadHelper.ThreadStart (object) [0x0000f] in <c453bf849bb84e719dbec0475e99db95>:0
at (wrapper runtime-invoke) <Module>.runtime_invoke_void__this___object (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0
"Thread Pool Worker" at <unknown> <0xffffffff>
at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0
... a lot more rows ...
Running Execution Dispatcher...
Is there a way to attach a complete log file? this forum will only allow 32000 chars :-)
It is still an issue, tested to update the host Ubuntu to the latest 18.04 packages to see if something gets better from that (including a later docker).
@atripp Yes, we are running nginx. But I will look into the keep-alive connections and see if that can give some hint on the problem.
We run prometheus to monitor our servers and cadvisor to export docker information to prometheus. It would be very useful if ProGet could export some sort of statistics to better monitor the load on the servers and maybe indicate the actual load and when you should upgrade to other licenses.
UPDATE in 2023 - see Inedo Products and Prometheus support to learn more about how to integrate our products together
@atripp Is there an easy way to see the number of ProGet requests?
I have tested to run tcpdump to find out some network-related information, but it's not that easy to determine how many simultaneous calls it actually serves.
Looking at the mono process it grows to more than 2 GB Resident memory on one day. Is there any number to compare this to?
@atripp If it would be a network related problem, wouldn't this error go away when the load lightens up? As we experience it there isn't much load on the server and when it starts to timeout on connections it won't recover until we restart the container. And when we do it justs work again.
We have added some monitoring with Prometheus (advisor for Docker and node_exporter for the host) and those reports don't show any significant load/traffic on the server when this happens.
Restarting container will make everything behave as "normal", querying the default NuGet feed in the UI takes approx 2 secs.
Hi
We are running ProGet (2.5.30 build 4) in docker on a Linux host, with another container for MSSQL. Our current default feed directed for Nuget often (once a week:ish) get very slow and if you view it in the web interface you will get an error on top.
"There was an error with a connector: The operation has timed out."
And in admin UI we can see:
Request URL: https://nuget.org/api/v2/Search()?$top=25&$filter=IsAbsoluteLatestVersion&searchTerm=%27%27&targetFramework=%27%27&includePrerelease=true&semVerLevel=2.0.0
System.Net.WebException: The operation has timed out.
at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task`1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func`1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.HttpWebRequest.EndGetResponse (System.IAsyncResult asyncResult) [0x00020] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Threading.Tasks.TaskFactory`1[TResult].FromAsyncCoreLogic (System.IAsyncResult iar, System.Func`2[T,TResult] endFunction, System.Action`1[T] endAction, System.Threading.Tasks.Task`1[TResult] promise, System.Boolean requiresSynchronization) [0x0000f] in <c453bf849bb84e719dbec0475e99db95>:0
--- End of stack trace from previous location where exception was thrown ---
at Inedo.ProGet.Feeds.StrongConnector.GetDirectResponseAsync (System.String url) [0x000eb] in <dacb7de90ce24964b2da21b67ac5e1d1>:0
at Inedo.ProGet.Feeds.StrongConnector.GetCachedResponseAsync (System.String url) [0x000a8] in <dacb7de90ce24964b2da21b67ac5e1d1>:0
at Inedo.ProGet.Feeds.StrongConnector.GetCachedTextReaderAsync (System.String url) [0x00075] in <dacb7de90ce24964b2da21b67ac5e1d1>:0
at Inedo.ProGet.Feeds.NuGet.NuGetFeedClient.PerformRequestWithAbsoluteUrlAsync (System.String fullUrl) [0x000d9] in <dacb7de90ce24964b2da21b67ac5e1d1>:0
Actual host and container show very little activity and we can curl the requested URL both from the docker host and from within the container without delays.
And once it has started to behave like this, it will not get better until we restart the container.
We currently restart our container once a day due to a potential memory leak described in another issue.
--Magnus
Thanks for the answer!
@atripp said in ProGet in docker/linux hanging after using all memory:
Hello; we haven't had any other users report this, so I'm afraid I don't have any idea how to help. It certainly sounds like a memory leak, and it'st most definitely a
mono-specific bug; unfortunately these are extremely hard to track down, and sometimes are even platform specific (i.e. depending on host operating system).I would make sure to upgrade to latest version of container image.
If you're not already using SQL Server for Linux (and you're using Postgres), then switch to SQL Server.
We use SQL Server.
I would simplify configuration; if you have a lot of connectors, etc.
We have 3 feeds and 2 connectors.
Try putting a ngix proxy infront of it.
We have that! =).
Once we have a clue about where the mono-bug is, we can at least consider ways to work-around it.
Yes. Maybe someone else have clues or have seen something similar. Most "linux container progets" run mono.exe right?
BR
Hello.
We run proget in a container on linux. We experience that the proget container consumes more and more memory (looking at the mono.exe process it runs) and then finally freezes once all memory on host machine is consumed. Only solution then is to restart the container (not the DB container, which is running fine). We have a 8GB host linux machine. Almost all of that memory is consumed by the proget container.
It really does seem like a memory leak. We are currently, using crontab, restarting the container every day as a workaround.
Do you have any idea what could be causing this and how we can proceed in troubleshooting it?