1. Home
  2. Categories
  3. Support
  4. Connector errors, timeout, restart needed.

Welcome to the Inedo Forums! Check out the Forums Guide for help getting started.

If you are experiencing any issues with the forum software, please visit the Contact Form on our website and let us know!

Connector errors, timeout, restart needed.

  • P

    Hi

    We are running ProGet (2.5.30 build 4) in docker on a Linux host, with another container for MSSQL. Our current default feed directed for Nuget often (once a week:ish) get very slow and if you view it in the web interface you will get an error on top.
    "There was an error with a connector: The operation has timed out."

    And in admin UI we can see:

    Request URL: https://nuget.org/api/v2/Search()?$top=25&$filter=IsAbsoluteLatestVersion&searchTerm=%27%27&targetFramework=%27%27&includePrerelease=true&semVerLevel=2.0.0

System.Net.WebException: The operation has timed out.
at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task`1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func`1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.HttpWebRequest.EndGetResponse (System.IAsyncResult asyncResult) [0x00020] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Threading.Tasks.TaskFactory`1[TResult].FromAsyncCoreLogic (System.IAsyncResult iar, System.Func`2[T,TResult] endFunction, System.Action`1[T] endAction, System.Threading.Tasks.Task`1[TResult] promise, System.Boolean requiresSynchronization) [0x0000f] in <c453bf849bb84e719dbec0475e99db95>:0
--- End of stack trace from previous location where exception was thrown ---

at Inedo.ProGet.Feeds.StrongConnector.GetDirectResponseAsync (System.String url) [0x000eb] in <dacb7de90ce24964b2da21b67ac5e1d1>:0
at Inedo.ProGet.Feeds.StrongConnector.GetCachedResponseAsync (System.String url) [0x000a8] in <dacb7de90ce24964b2da21b67ac5e1d1>:0
at Inedo.ProGet.Feeds.StrongConnector.GetCachedTextReaderAsync (System.String url) [0x00075] in <dacb7de90ce24964b2da21b67ac5e1d1>:0
at Inedo.ProGet.Feeds.NuGet.NuGetFeedClient.PerformRequestWithAbsoluteUrlAsync (System.String fullUrl) [0x000d9] in <dacb7de90ce24964b2da21b67ac5e1d1>:0

    Actual host and container show very little activity and we can curl the requested URL both from the docker host and from within the container without delays.
    And once it has started to behave like this, it will not get better until we restart the container.
    We currently restart our container once a day due to a potential memory leak described in another issue.

    --Magnus

    0 P 2 Replies
  • P

    This post is deleted!
    0
  • P

    Restarting container will make everything behave as "normal", querying the default NuGet feed in the UI takes approx 2 secs.

    0
  • atripp
    inedo-engineer

    This sounds like it's network-traffic related, and likely the server is getting overwhelmed...

    The ProGet Free to ProGet Enterprise blog article talks more about this, but ultimately load-balancing is a solution that will prevent this peak usage / overwhelming problem.

    0 P 2 Replies
  • P

    @atripp If it would be a network related problem, wouldn't this error go away when the load lightens up? As we experience it there isn't much load on the server and when it starts to timeout on connections it won't recover until we restart the container. And when we do it justs work again.

    We have added some monitoring with Prometheus (advisor for Docker and node_exporter for the host) and those reports don't show any significant load/traffic on the server when this happens.

    0
  • P

    @atripp Is there an easy way to see the number of ProGet requests?

    I have tested to run tcpdump to find out some network-related information, but it's not that easy to determine how many simultaneous calls it actually serves.

    0
  • atripp
    inedo-engineer

    I'm not aware of how to do it; ProGet does not provide monitoring at this network level. Do you have an ngix proxy in front of it? We've heard that's very important, and it can help deal with some other mono bugs around not closing keep-alive connections quickly. And and also lets you have SSL as well.

    0 P 1 Reply
  • P

    @atripp Yes, we are running nginx. But I will look into the keep-alive connections and see if that can give some hint on the problem.

    0
  • P

    It is still an issue, tested to update the host Ubuntu to the latest 18.04 packages to see if something gets better from that (including a later docker).

    0
  • rhessinger
    inedo-engineer

    Hi @patchlings,

    Have you enabled the nginx HTTP Stub Status module on your nginx proxy? I would be interested to see the number of open and waiting connections. Although this article is old, nginx: See Active connections / Connections Per Seconds seems to show a pretty easy configuration to see this status. I would be curious to see these counts when the problem exists. Would you be able to enable this and post the counts when everything is working and when it is not?

    Thanks,
    Rich

    0
  • P

    @rhessinger I will look into that!

    Meanwhile, I tested to run "kill -QUIT $MONO_PID" to get some thread dumps from the mono process when the problem appears.

    "Thread Pool Worker"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0

"Thread Pool Worker"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0

"Thread Pool Worker"
"Thread Pool Worker"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0

"Thread Pool Worker"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0

"Thread Pool Worker"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0

"Thread Pool Worker"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0

"Thread Pool I/O Selector"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) System.Threading.ThreadPool.RequestWorkerThread () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadPoolWorkQueue.EnsureThreadRequested () [0x0001f] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadPoolWorkQueue.Enqueue (System.Threading.IThreadPoolWorkItem,bool) [0x00071] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadPool.UnsafeQueueCustomWorkItem (System.Threading.IThreadPoolWorkItem,bool) [0x00007] in <c453bf849bb84e719dbec0475e99db95>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_void_object_byte (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0

"Thread Pool Worker"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0

"Thread Pool Worker"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) System.Net.Sockets.Socket.Receive_internal (intptr,byte*,int,System.Net.Sockets.SocketFlags,int&,bool) [0x00000] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.Sockets.Socket.Receive_internal (System.Net.Sockets.SafeSocketHandle,byte*,int,System.Net.Sockets.SocketFlags,int&,bool) [0x0000c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.Sockets.Socket.Receive (byte[],int,int,System.Net.Sockets.SocketFlags,System.Net.Sockets.SocketError&) [0x0002e] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.Sockets.Socket.Receive (byte[],int,int,System.Net.Sockets.SocketFlags) [0x00000] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.Sockets.NetworkStream.Read (byte[],int,int) [0x00065] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Data.SqlClient.SNI.SslOverTdsStream/<ReadInternal>d__11.MoveNext () [0x000e8] in <0864334e7e474248b37afca9b637daa9>:0
  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1<int>.Start<System.Data.SqlClient.SNI.SslOverTdsStream/<ReadInternal>d__11> (System.Data.SqlClient.SNI.SslOverTdsStream/<ReadInternal>d__11&) [0x0002c] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Data.SqlClient.SNI.SslOverTdsStream.ReadInternal (byte[],int,int,System.Threading.CancellationToken,bool) [0x0004d] in <0864334e7e474248b37afca9b637daa9>:0
  at System.Data.SqlClient.SNI.SslOverTdsStream.Read (byte[],int,int) [0x00009] in <0864334e7e474248b37afca9b637daa9>:0
  at Mono.Net.Security.MobileAuthenticatedStream/<>c__DisplayClass66_0.<InnerRead>b__0 () [0x0002b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Threading.Tasks.Task`1<int>.InnerInvoke () [0x0000f] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.Tasks.Task.Execute () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.Tasks.Task.ExecutionContextCallback (object) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00071] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.Tasks.Task.ExecuteWithThreadLocal (System.Threading.Tasks.Task&) [0x00034] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.Tasks.Task.ExecuteEntry (bool) [0x0004a] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.Tasks.Task.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0

"Thread Pool Worker"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0

"Timer-Scheduler"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) System.Threading.WaitHandle.Wait_internal (intptr*,int,bool,int) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.WaitHandle.WaitOneNative (System.Runtime.InteropServices.SafeHandle,uint,bool,bool) [0x00044] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.WaitHandle.InternalWaitOne (System.Runtime.InteropServices.SafeHandle,long,bool,bool) [0x00014] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.WaitHandle.WaitOne (long,bool) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.WaitHandle.WaitOne (int,bool) [0x00019] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.WaitHandle.WaitOne (int) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.Timer/Scheduler.SchedulerThread () [0x0003c] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadHelper.ThreadStart_Context (object) [0x00014] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00071] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object) [0x0002b] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadHelper.ThreadStart () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
  at (wrapper runtime-invoke) object.runtime_invoke_void__this__ (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0

"Thread Pool Worker"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0

"<unnamed thread>"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) Interop/Sys.Read (intptr,byte*,int) [0x00000] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at Interop/Sys.Read (Microsoft.Win32.SafeHandles.SafeFileHandle,byte*,int) [0x00012] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.IO.CoreFX.FileSystemWatcher/RunningInstance.TryReadEvent (System.IO.CoreFX.FileSystemWatcher/RunningInstance/NotifyEvent&) [0x00027] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.IO.CoreFX.FileSystemWatcher/RunningInstance.ProcessEvents () [0x002dc] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.IO.CoreFX.FileSystemWatcher/RunningInstance/<>c.<Start>b__14_0 (object) [0x00000] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Threading.Tasks.Task.InnerInvoke () [0x00025] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.Tasks.Task.Execute () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.Tasks.Task.ExecutionContextCallback (object) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00071] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.Tasks.Task.ExecuteWithThreadLocal (System.Threading.Tasks.Task&) [0x00034] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.Tasks.Task.ExecuteEntry (bool) [0x0004a] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.Tasks.ThreadPoolTaskScheduler/<>c.<.cctor>b__10_0 (object) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadHelper.ThreadStart_Context (object) [0x00025] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00071] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object) [0x0002b] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadHelper.ThreadStart (object) [0x0000f] in <c453bf849bb84e719dbec0475e99db95>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_void__this___object (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0

"Thread Pool Worker"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0

"Thread Pool Worker"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0

"Thread Pool Worker"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0

"Thread Pool Worker"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0

"Thread Pool Worker"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0

"Thread Pool Worker"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0

"Thread Pool Worker"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0

"Finalizer"
"<unnamed thread>"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) Interop/Sys.Read (intptr,byte*,int) [0x00000] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at Interop/Sys.Read (Microsoft.Win32.SafeHandles.SafeFileHandle,byte*,int) [0x00012] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.IO.CoreFX.FileSystemWatcher/RunningInstance.TryReadEvent (System.IO.CoreFX.FileSystemWatcher/RunningInstance/NotifyEvent&) [0x00027] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.IO.CoreFX.FileSystemWatcher/RunningInstance.ProcessEvents () [0x002dc] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.IO.CoreFX.FileSystemWatcher/RunningInstance/<>c.<Start>b__14_0 (object) [0x00000] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Threading.Tasks.Task.InnerInvoke () [0x00025] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.Tasks.Task.Execute () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.Tasks.Task.ExecutionContextCallback (object) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00071] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.Tasks.Task.ExecuteWithThreadLocal (System.Threading.Tasks.Task&) [0x00034] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.Tasks.Task.ExecuteEntry (bool) [0x0004a] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.Tasks.ThreadPoolTaskScheduler/<>c.<.cctor>b__10_0 (object) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadHelper.ThreadStart_Context (object) [0x00025] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00071] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object) [0x0002b] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadHelper.ThreadStart (object) [0x0000f] in <c453bf849bb84e719dbec0475e99db95>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_void__this___object (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0

"Thread Pool Worker"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0

... a lot more rows ...

Running Execution Dispatcher...

    Is there a way to attach a complete log file? this forum will only allow 32000 chars :-)

    0
  • rhessinger
    inedo-engineer

    Hi @patchlings ,

    I was doing some Docker research around this issue. Something I found Docker users complaining about was an issue with the total number of available connections on the host. It seems that the more containers running on a host (especially with database containers on it), the connection count can get spread a bit thin under load. Have you tried increasing the max number of connections? Here is a StackOverflow about increasing them Increasing the maximum number of TCP/IP connections in Linux.

    Also, how many connectors do you have on your NuGet feed?

    Thanks,
    Rich

    0 P 1 Reply
  • P

    @rhessinger We have enabled the stub_status for Nginx, when the problem appeared just now the status showed the following page:

    Active connections: 15
    server accepts handled requests
    428494 428494 756561
    Reading: 0 Writing: 2 Waiting: 13

    We have some network connection monitoring with prometheus for the host and they show nothing specific (see attached image)connections.png

    0
  • P

    @rhessinger said in Connector errors, timeout, restart needed.:

    Also, how many connectors do you have on your NuGet feed?

    Sorry, I missed this question... We have four connectors (but only uses three of them): connectors.png

    And when I looked that up I noticed that the one we don't use was marked with error, removed it.

    0
  • rhessinger
    inedo-engineer

    Hi @patchlings,

    Do these connector errors happen on all your feeds? Or just NuGet?

    Thanks,
    Rich

    0 P 1 Reply
  • P

    @rhessinger 99% NuGet connector errors.

    0
  • rhessinger
    inedo-engineer

    Hi @patchlings,

    Thank you very much for the information. I'm researching the issue more on our side. I may send over some more questions your way while I'm working on this.

    Thanks,
    Rich

    0
  • P

    Now it's seems flaky again:
    nginx status:
    Active connections: 57
    server accepts handled requests
    462208 462208 840656
    Reading: 0 Writing: 30 Waiting: 27

    Main web UI can't be reached at all.
    Ran sudo kill -QUIT $MONO_PID and got myself a huge thread dump with a lot of entries like this (let me know if you want me to supply the entire dump, (it's more then the allowed 32k chars):

    "Thread Pool Worker"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0
    0
  • P

    One interesting observation is that when the system "freezes" there seems to be no traffic at all (12:30):
    f64b41a1-47e2-4f91-8c61-3fff283a79b3-image.png

    0
  • P

    Performing strace on the mono process gives a lot of:

    [pid 26706] recvfrom(39, "", 8, 0, NULL, NULL) = 0
[pid 26706] recvfrom(39, "", 8, 0, NULL, NULL) = 0
[pid 26706] recvfrom(39, "", 8, 0, NULL, NULL) = 0
[pid 26706] recvfrom(39, "", 8, 0, NULL, NULL) = 0
[pid 26706] recvfrom(39, "", 8, 0, NULL, NULL) = 0
[pid 26706] recvfrom(39, "", 8, 0, NULL, NULL) = 0
[pid 26706] recvfrom(39, "", 8, 0, NULL, NULL) = 0

    Then I found this: https://stackoverflow.com/questions/24000534/tracing-socket-file-descriptor-back-to-bound-address

    So the next step will be to see if I can find out which socket that is actually appearing in the strace.

    0
  • rhessinger
    inedo-engineer

    Hi @patchlings ,

    Thanks for all this information. Would you be able to get me an open thread count for when everything is running fine and when things are not? You should be able to find it by running sudo ps huH p <PID_OF_MONO_PROCESS> | wc -l. Also, are you running these commands inside the docker image? Or on the host?

    Thanks,
    Rich

    0
  • P

    @rhessinger Hi, if I run it on the host (or in the actual container) it seems to vary between 25-45, haven't seem anything different between when it behaves bad:

    ubuntu@proget:/opt/provision/additional$ sudo ps huH p 12436 | wc -l
29
ubuntu@proget:/opt/provision/additional$ sudo ps huH p 12436 | wc -l
44
    0
  • rhessinger
    inedo-engineer

    Hi @patchlings,

    Thank you for checking this for me. So the good thing is that this is definitely not a threading issue. Can you give me a stub status when it's working during normal use and one after it starts behaving poorly?

    Thanks,
    Rich

    0 P 1 Reply
  • P

    @rhessinger I will do that.

    One note, it seems that running on debug logging level makes it a little more unstable. We are back to normal "warning" now and now it seems to be back at our regular "once a day" frequency. With debug it felt like 5 times a day.

    0
  • rhessinger
    inedo-engineer

    Hi @patchlings,

    Thanks! I'm not surprised that debug exaccerbates the issue, but it's definitely good to know.

    Thanks,
    Rich

    0
  • P

    Now it misbehaves...

    Active connections: 6 
server accepts handled requests
 115659 115659 179524 
Reading: 0 Writing: 1 Waiting: 5

    Accessing feeds/default will show a "There was an error with a connector: The operation has timed out." All other feeds work as expected.

    ubuntu@proget:~$ sudo ps huH p 18819 |wc -l
33

    In the admin interface:

    Request URL: https://nuget.org/api/v2/Search()?$top=25&$filter=IsAbsoluteLatestVersion&searchTerm=%27%27&targetFramework=%27%27&includePrerelease=true&semVerLevel=2.0.0

System.Net.WebException: The operation has timed out.
at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task`1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func`1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.HttpWebRequest.EndGetResponse (System.IAsyncResult asyncResult) [0x00020] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Threading.Tasks.TaskFactory`1[TResult].FromAsyncCoreLogic (System.IAsyncResult iar, System.Func`2[T,TResult] endFunction, System.Action`1[T] endAction, System.Threading.Tasks.Task`1[TResult] promise, System.Boolean requiresSynchronization) [0x0000f] in <c453bf849bb84e719dbec0475e99db95>:0
--- End of stack trace from previous location where exception was thrown ---

at Inedo.ProGet.Feeds.StrongConnector.GetDirectResponseAsync (System.String url) [0x000eb] in <dacb7de90ce24964b2da21b67ac5e1d1>:0
at Inedo.ProGet.Feeds.StrongConnector.GetCachedResponseAsync (System.String url) [0x000a8] in <dacb7de90ce24964b2da21b67ac5e1d1>:0
at Inedo.ProGet.Feeds.StrongConnector.GetCachedTextReaderAsync (System.String url) [0x00075] in <dacb7de90ce24964b2da21b67ac5e1d1>:0
at Inedo.ProGet.Feeds.NuGet.NuGetFeedClient.PerformRequestWithAbsoluteUrlAsync (System.String fullUrl) [0x000d9] in <dacb7de90ce24964b2da21b67ac5e1d1>:0

    strace doesn't show anything obvious.

    Restart docker container and now the /feed/default shows no problem. nginx status:

    Active connections: 6 
server accepts handled requests
 115829 115829 180229 
Reading: 0 Writing: 1 Waiting: 5
    0
  • rhessinger
    inedo-engineer

    Hi @patchlings,

    Thanks for all the data. I really appreciate your help with this! I'm currently reviewing our code to see if I can find a correlation with this.

    Thanks,
    Rich

    0
  • rhessinger
    inedo-engineer

    Hi @patchlings,

    We believe this error is due to a regression/bug in the Mono, which is related to change to their HttpWebRequest stack. In the future, after Microsoft releases it, we plan to move to .NET 5 for a more stable platform.

    The best resolution, for now, is to move to Windows/IIS; it's the most stable. Is this possible?

    If not... we don't know when Mono will fix the bug, or if it's possible for us to work-around, but we think it will get resolved in the future. Would you be able to make ProGet restart once per day? I think this would help with the memory leak.

    Thanks,
    Rich

    0
  • P

    Hi @rhessinger

    Ok, the issue you found sounds interesting, but it seems to be more connected to the other problem we have: https://forums.inedo.com/topic/2887/proget-in-docker-linux-hanging-after-using-all-memory/6

    And we are currently already restarting the docker container once a day to make sure that we don't run out of memory.

    And looking into the memory graphs it looks like we experience the connection issue even if the memory consumption looks ok (Green bars = scheduled restarts, red bars = connector problems, yellow graph = available memory, green graph = free memory): memory.png

    -- Magnus

    0
  • rhessinger
    inedo-engineer

    Hi @patchlings ,

    That is the heart of the Mono bug that I referenced. Currently Mono has a memory leak around HTTP requests which will cause our connectors to hang when the memory usage is too high. They have reduced the leaking substantially starting in Mono 5.16, but there is still a leak. If you search around on this error, you will find similar questions with almost identical errors still in 2020. Unfortunately, we are in the hands of the Mono team until we switch off Mono and migrate to .NET 5.

    Thanks,
    Rich

    0
  • P

    @rhessinger Ok, then I understand. We will look into our options then. Depending on Mono fixing this seems like a bad solution as it seems to have appeared for quite some time.

    Thanks for all your help!

    What is the timeline of your Mono to .NET 5 conversion?

    0
  • rhessinger
    inedo-engineer

    Hi @patchlings,

    No problem! We are still investigating some workarounds, but we don't have anything specific yet.

    We don't currently have a timeline for the .NET 5 switch. We have attempted a .NET Core transition in early 2019, but we ran into too many instability issues and scrapped it. We will wait for .NET 5 to be officially released (Microsoft says November 2020) before we attempt the switch, but it will all come down to the stability of ProGet on .NET 5.

    Thanks,
    Rich

    0
Log in to reply
 
Inedo Website Home • Support Home • Code of Conduct • Forums Guide • Documentation