Inedo Community Forums Forums
    • Recent
    • Tags
    • Popular
    • Login

    Welcome to the Inedo Forums! Check out the Forums Guide for help getting started.

    If you are experiencing any issues with the forum software, please visit the Contact Form on our website and let us know!

    Connector errors, timeout, restart needed.

    Scheduled Pinned Locked Moved Support
    32 Posts 3 Posters 76 Views 1 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • P Offline
      patchlings @patchlings
      last edited by

      This post is deleted!
      1 Reply Last reply Reply Quote 0
      • P Offline
        patchlings @patchlings
        last edited by

        Restarting container will make everything behave as "normal", querying the default NuGet feed in the UI takes approx 2 secs.

        1 Reply Last reply Reply Quote 0
        • atrippA Offline
          atripp inedo-engineer
          last edited by apxltd

          This sounds like it's network-traffic related, and likely the server is getting overwhelmed...

          See How to Prevent Server Overload in ProGet to learn more.

          P 2 Replies Last reply Reply Quote 0
          • P Offline
            patchlings @atripp
            last edited by

            @atripp If it would be a network related problem, wouldn't this error go away when the load lightens up? As we experience it there isn't much load on the server and when it starts to timeout on connections it won't recover until we restart the container. And when we do it justs work again.

            We have added some monitoring with Prometheus (advisor for Docker and node_exporter for the host) and those reports don't show any significant load/traffic on the server when this happens.

            1 Reply Last reply Reply Quote 0
            • P Offline
              patchlings @atripp
              last edited by

              @atripp Is there an easy way to see the number of ProGet requests?

              I have tested to run tcpdump to find out some network-related information, but it's not that easy to determine how many simultaneous calls it actually serves.

              1 Reply Last reply Reply Quote 0
              • atrippA Offline
                atripp inedo-engineer
                last edited by

                I'm not aware of how to do it; ProGet does not provide monitoring at this network level. Do you have an ngix proxy in front of it? We've heard that's very important, and it can help deal with some other mono bugs around not closing keep-alive connections quickly. And and also lets you have SSL as well.

                P 1 Reply Last reply Reply Quote 0
                • P Offline
                  patchlings @atripp
                  last edited by

                  @atripp Yes, we are running nginx. But I will look into the keep-alive connections and see if that can give some hint on the problem.

                  1 Reply Last reply Reply Quote 0
                  • P Offline
                    patchlings
                    last edited by

                    It is still an issue, tested to update the host Ubuntu to the latest 18.04 packages to see if something gets better from that (including a later docker).

                    1 Reply Last reply Reply Quote 0
                    • rhessingerR Offline
                      rhessinger inedo-engineer
                      last edited by

                      Hi @patchlings,

                      Have you enabled the nginx HTTP Stub Status module on your nginx proxy? I would be interested to see the number of open and waiting connections. Although this article is old, nginx: See Active connections / Connections Per Seconds seems to show a pretty easy configuration to see this status. I would be curious to see these counts when the problem exists. Would you be able to enable this and post the counts when everything is working and when it is not?

                      Thanks,
                      Rich

                      Products Engineer, Inedo

                      1 Reply Last reply Reply Quote 0
                      • P Offline
                        patchlings
                        last edited by

                        @rhessinger I will look into that!

                        Meanwhile, I tested to run "kill -QUIT $MONO_PID" to get some thread dumps from the mono process when the problem appears.

                        "Thread Pool Worker"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0

"Thread Pool Worker"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0

"Thread Pool Worker"
"Thread Pool Worker"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0

"Thread Pool Worker"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0

"Thread Pool Worker"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0

"Thread Pool Worker"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0

"Thread Pool I/O Selector"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) System.Threading.ThreadPool.RequestWorkerThread () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadPoolWorkQueue.EnsureThreadRequested () [0x0001f] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadPoolWorkQueue.Enqueue (System.Threading.IThreadPoolWorkItem,bool) [0x00071] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadPool.UnsafeQueueCustomWorkItem (System.Threading.IThreadPoolWorkItem,bool) [0x00007] in <c453bf849bb84e719dbec0475e99db95>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_void_object_byte (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0

"Thread Pool Worker"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0

"Thread Pool Worker"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) System.Net.Sockets.Socket.Receive_internal (intptr,byte*,int,System.Net.Sockets.SocketFlags,int&,bool) [0x00000] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.Sockets.Socket.Receive_internal (System.Net.Sockets.SafeSocketHandle,byte*,int,System.Net.Sockets.SocketFlags,int&,bool) [0x0000c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.Sockets.Socket.Receive (byte[],int,int,System.Net.Sockets.SocketFlags,System.Net.Sockets.SocketError&) [0x0002e] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.Sockets.Socket.Receive (byte[],int,int,System.Net.Sockets.SocketFlags) [0x00000] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.Sockets.NetworkStream.Read (byte[],int,int) [0x00065] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Data.SqlClient.SNI.SslOverTdsStream/<ReadInternal>d__11.MoveNext () [0x000e8] in <0864334e7e474248b37afca9b637daa9>:0
  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1<int>.Start<System.Data.SqlClient.SNI.SslOverTdsStream/<ReadInternal>d__11> (System.Data.SqlClient.SNI.SslOverTdsStream/<ReadInternal>d__11&) [0x0002c] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Data.SqlClient.SNI.SslOverTdsStream.ReadInternal (byte[],int,int,System.Threading.CancellationToken,bool) [0x0004d] in <0864334e7e474248b37afca9b637daa9>:0
  at System.Data.SqlClient.SNI.SslOverTdsStream.Read (byte[],int,int) [0x00009] in <0864334e7e474248b37afca9b637daa9>:0
  at Mono.Net.Security.MobileAuthenticatedStream/<>c__DisplayClass66_0.<InnerRead>b__0 () [0x0002b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Threading.Tasks.Task`1<int>.InnerInvoke () [0x0000f] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.Tasks.Task.Execute () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.Tasks.Task.ExecutionContextCallback (object) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00071] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.Tasks.Task.ExecuteWithThreadLocal (System.Threading.Tasks.Task&) [0x00034] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.Tasks.Task.ExecuteEntry (bool) [0x0004a] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.Tasks.Task.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0

"Thread Pool Worker"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0

"Timer-Scheduler"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) System.Threading.WaitHandle.Wait_internal (intptr*,int,bool,int) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.WaitHandle.WaitOneNative (System.Runtime.InteropServices.SafeHandle,uint,bool,bool) [0x00044] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.WaitHandle.InternalWaitOne (System.Runtime.InteropServices.SafeHandle,long,bool,bool) [0x00014] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.WaitHandle.WaitOne (long,bool) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.WaitHandle.WaitOne (int,bool) [0x00019] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.WaitHandle.WaitOne (int) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.Timer/Scheduler.SchedulerThread () [0x0003c] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadHelper.ThreadStart_Context (object) [0x00014] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00071] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object) [0x0002b] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadHelper.ThreadStart () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
  at (wrapper runtime-invoke) object.runtime_invoke_void__this__ (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0

"Thread Pool Worker"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0

"<unnamed thread>"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) Interop/Sys.Read (intptr,byte*,int) [0x00000] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at Interop/Sys.Read (Microsoft.Win32.SafeHandles.SafeFileHandle,byte*,int) [0x00012] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.IO.CoreFX.FileSystemWatcher/RunningInstance.TryReadEvent (System.IO.CoreFX.FileSystemWatcher/RunningInstance/NotifyEvent&) [0x00027] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.IO.CoreFX.FileSystemWatcher/RunningInstance.ProcessEvents () [0x002dc] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.IO.CoreFX.FileSystemWatcher/RunningInstance/<>c.<Start>b__14_0 (object) [0x00000] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Threading.Tasks.Task.InnerInvoke () [0x00025] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.Tasks.Task.Execute () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.Tasks.Task.ExecutionContextCallback (object) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00071] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.Tasks.Task.ExecuteWithThreadLocal (System.Threading.Tasks.Task&) [0x00034] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.Tasks.Task.ExecuteEntry (bool) [0x0004a] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.Tasks.ThreadPoolTaskScheduler/<>c.<.cctor>b__10_0 (object) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadHelper.ThreadStart_Context (object) [0x00025] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00071] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object) [0x0002b] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadHelper.ThreadStart (object) [0x0000f] in <c453bf849bb84e719dbec0475e99db95>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_void__this___object (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0

"Thread Pool Worker"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0

"Thread Pool Worker"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0

"Thread Pool Worker"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0

"Thread Pool Worker"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0

"Thread Pool Worker"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0

"Thread Pool Worker"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0

"Thread Pool Worker"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0

"Finalizer"
"<unnamed thread>"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) Interop/Sys.Read (intptr,byte*,int) [0x00000] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at Interop/Sys.Read (Microsoft.Win32.SafeHandles.SafeFileHandle,byte*,int) [0x00012] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.IO.CoreFX.FileSystemWatcher/RunningInstance.TryReadEvent (System.IO.CoreFX.FileSystemWatcher/RunningInstance/NotifyEvent&) [0x00027] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.IO.CoreFX.FileSystemWatcher/RunningInstance.ProcessEvents () [0x002dc] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.IO.CoreFX.FileSystemWatcher/RunningInstance/<>c.<Start>b__14_0 (object) [0x00000] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Threading.Tasks.Task.InnerInvoke () [0x00025] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.Tasks.Task.Execute () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.Tasks.Task.ExecutionContextCallback (object) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00071] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.Tasks.Task.ExecuteWithThreadLocal (System.Threading.Tasks.Task&) [0x00034] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.Tasks.Task.ExecuteEntry (bool) [0x0004a] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.Tasks.ThreadPoolTaskScheduler/<>c.<.cctor>b__10_0 (object) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadHelper.ThreadStart_Context (object) [0x00025] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00071] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object) [0x0002b] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadHelper.ThreadStart (object) [0x0000f] in <c453bf849bb84e719dbec0475e99db95>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_void__this___object (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0

"Thread Pool Worker"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0

... a lot more rows ...

Running Execution Dispatcher...

                        Is there a way to attach a complete log file? this forum will only allow 32000 chars :-)

                        1 Reply Last reply Reply Quote 0
                        • rhessingerR Offline
                          rhessinger inedo-engineer
                          last edited by

                          Hi @patchlings ,

                          I was doing some Docker research around this issue. Something I found Docker users complaining about was an issue with the total number of available connections on the host. It seems that the more containers running on a host (especially with database containers on it), the connection count can get spread a bit thin under load. Have you tried increasing the max number of connections? Here is a StackOverflow about increasing them Increasing the maximum number of TCP/IP connections in Linux.

                          Also, how many connectors do you have on your NuGet feed?

                          Thanks,
                          Rich

                          Products Engineer, Inedo

                          P 1 Reply Last reply Reply Quote 0
                          • P Offline
                            patchlings @rhessinger
                            last edited by

                            @rhessinger We have enabled the stub_status for Nginx, when the problem appeared just now the status showed the following page:

                            Active connections: 15
                            server accepts handled requests
                            428494 428494 756561
                            Reading: 0 Writing: 2 Waiting: 13

                            We have some network connection monitoring with prometheus for the host and they show nothing specific (see attached image)connections.png

                            1 Reply Last reply Reply Quote 0
                            • P Offline
                              patchlings
                              last edited by patchlings

                              @rhessinger said in Connector errors, timeout, restart needed.:

                              Also, how many connectors do you have on your NuGet feed?

                              Sorry, I missed this question... We have four connectors (but only uses three of them): connectors.png

                              And when I looked that up I noticed that the one we don't use was marked with error, removed it.

                              1 Reply Last reply Reply Quote 0
                              • rhessingerR Offline
                                rhessinger inedo-engineer
                                last edited by

                                Hi @patchlings,

                                Do these connector errors happen on all your feeds? Or just NuGet?

                                Thanks,
                                Rich

                                Products Engineer, Inedo

                                P 1 Reply Last reply Reply Quote 0
                                • P Offline
                                  patchlings @rhessinger
                                  last edited by

                                  @rhessinger 99% NuGet connector errors.

                                  1 Reply Last reply Reply Quote 0
                                  • rhessingerR Offline
                                    rhessinger inedo-engineer
                                    last edited by

                                    Hi @patchlings,

                                    Thank you very much for the information. I'm researching the issue more on our side. I may send over some more questions your way while I'm working on this.

                                    Thanks,
                                    Rich

                                    Products Engineer, Inedo

                                    1 Reply Last reply Reply Quote 0
                                    • P Offline
                                      patchlings
                                      last edited by

                                      Now it's seems flaky again:
                                      nginx status:
                                      Active connections: 57
                                      server accepts handled requests
                                      462208 462208 840656
                                      Reading: 0 Writing: 30 Waiting: 27

                                      Main web UI can't be reached at all.
                                      Ran sudo kill -QUIT $MONO_PID and got myself a huge thread dump with a lot of entries like this (let me know if you want me to supply the entire dump, (it's more then the allowed 32k chars):

                                      "Thread Pool Worker"  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
  at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
  at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0
                                      1 Reply Last reply Reply Quote 0
                                      • P Offline
                                        patchlings
                                        last edited by

                                        One interesting observation is that when the system "freezes" there seems to be no traffic at all (12:30):
                                        f64b41a1-47e2-4f91-8c61-3fff283a79b3-image.png

                                        1 Reply Last reply Reply Quote 0
                                        • P Offline
                                          patchlings
                                          last edited by

                                          Performing strace on the mono process gives a lot of:

                                          [pid 26706] recvfrom(39, "", 8, 0, NULL, NULL) = 0
[pid 26706] recvfrom(39, "", 8, 0, NULL, NULL) = 0
[pid 26706] recvfrom(39, "", 8, 0, NULL, NULL) = 0
[pid 26706] recvfrom(39, "", 8, 0, NULL, NULL) = 0
[pid 26706] recvfrom(39, "", 8, 0, NULL, NULL) = 0
[pid 26706] recvfrom(39, "", 8, 0, NULL, NULL) = 0
[pid 26706] recvfrom(39, "", 8, 0, NULL, NULL) = 0

                                          Then I found this: https://stackoverflow.com/questions/24000534/tracing-socket-file-descriptor-back-to-bound-address

                                          So the next step will be to see if I can find out which socket that is actually appearing in the strace.

                                          1 Reply Last reply Reply Quote 0
                                          • rhessingerR Offline
                                            rhessinger inedo-engineer
                                            last edited by

                                            Hi @patchlings ,

                                            Thanks for all this information. Would you be able to get me an open thread count for when everything is running fine and when things are not? You should be able to find it by running sudo ps huH p <PID_OF_MONO_PROCESS> | wc -l. Also, are you running these commands inside the docker image? Or on the host?

                                            Thanks,
                                            Rich

                                            Products Engineer, Inedo

                                            1 Reply Last reply Reply Quote 0

                                            Hello! It looks like you're interested in this conversation, but you don't have an account yet.

                                            Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.

                                            With your input, this post could be even better 💗

                                            Register Login
                                            • 1
                                            • 2
                                            • 1 / 2
                                            • First post
                                              Last post
                                            Inedo Website Home • Support Home • Code of Conduct • Forums Guide • Documentation