Navigation

    Inedo Community Forums

    Forums

    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    1. Home
    2. patchlings
    P
    • Profile
    • Following
    • Followers
    • Topics
    • Posts
    • Best
    • Groups

    patchlings

    @patchlings

    2
    Reputation
    23
    Posts
    1
    Profile views
    0
    Followers
    0
    Following
    Joined Last Online

    patchlings Follow

    Best posts made by patchlings

    • Prometheus support?

      We run prometheus to monitor our servers and cadvisor to export docker information to prometheus. It would be very useful if ProGet could export some sort of statistics to better monitor the load on the servers and maybe indicate the actual load and when you should upgrade to other licenses.

      UPDATE in 2023 - see Inedo Products and Prometheus support to learn more about how to integrate our products together

      posted in Support
      P
      patchlings

    Latest posts made by patchlings

    • RE: Connector errors, timeout, restart needed.

      @rhessinger Ok, then I understand. We will look into our options then. Depending on Mono fixing this seems like a bad solution as it seems to have appeared for quite some time.

      Thanks for all your help!

      What is the timeline of your Mono to .NET 5 conversion?

      posted in Support
      P
      patchlings
    • RE: Connector errors, timeout, restart needed.

      Hi @rhessinger

      Ok, the issue you found sounds interesting, but it seems to be more connected to the other problem we have: https://forums.inedo.com/topic/2887/proget-in-docker-linux-hanging-after-using-all-memory/6

      And we are currently already restarting the docker container once a day to make sure that we don't run out of memory.

      And looking into the memory graphs it looks like we experience the connection issue even if the memory consumption looks ok (Green bars = scheduled restarts, red bars = connector problems, yellow graph = available memory, green graph = free memory): memory.png

      -- Magnus

      posted in Support
      P
      patchlings
    • RE: Connector errors, timeout, restart needed.

      Now it misbehaves...

      Active connections: 6 
      server accepts handled requests
       115659 115659 179524 
      Reading: 0 Writing: 1 Waiting: 5
      

      Accessing feeds/default will show a "There was an error with a connector: The operation has timed out." All other feeds work as expected.

      ubuntu@proget:~$ sudo ps huH p 18819 |wc -l
      33
      

      In the admin interface:

      Request URL: https://nuget.org/api/v2/Search()?$top=25&$filter=IsAbsoluteLatestVersion&searchTerm=%27%27&targetFramework=%27%27&includePrerelease=true&semVerLevel=2.0.0
      
      System.Net.WebException: The operation has timed out.
      at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task`1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func`1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <4d8f95b7df544b3fa1a27dda924b4424>:0
      at System.Net.HttpWebRequest.EndGetResponse (System.IAsyncResult asyncResult) [0x00020] in <4d8f95b7df544b3fa1a27dda924b4424>:0
      at System.Threading.Tasks.TaskFactory`1[TResult].FromAsyncCoreLogic (System.IAsyncResult iar, System.Func`2[T,TResult] endFunction, System.Action`1[T] endAction, System.Threading.Tasks.Task`1[TResult] promise, System.Boolean requiresSynchronization) [0x0000f] in <c453bf849bb84e719dbec0475e99db95>:0
      --- End of stack trace from previous location where exception was thrown ---
      
      at Inedo.ProGet.Feeds.StrongConnector.GetDirectResponseAsync (System.String url) [0x000eb] in <dacb7de90ce24964b2da21b67ac5e1d1>:0
      at Inedo.ProGet.Feeds.StrongConnector.GetCachedResponseAsync (System.String url) [0x000a8] in <dacb7de90ce24964b2da21b67ac5e1d1>:0
      at Inedo.ProGet.Feeds.StrongConnector.GetCachedTextReaderAsync (System.String url) [0x00075] in <dacb7de90ce24964b2da21b67ac5e1d1>:0
      at Inedo.ProGet.Feeds.NuGet.NuGetFeedClient.PerformRequestWithAbsoluteUrlAsync (System.String fullUrl) [0x000d9] in <dacb7de90ce24964b2da21b67ac5e1d1>:0
      

      strace doesn't show anything obvious.

      Restart docker container and now the /feed/default shows no problem. nginx status:

      Active connections: 6 
      server accepts handled requests
       115829 115829 180229 
      Reading: 0 Writing: 1 Waiting: 5 
      
      posted in Support
      P
      patchlings
    • RE: Connector errors, timeout, restart needed.

      @rhessinger I will do that.

      One note, it seems that running on debug logging level makes it a little more unstable. We are back to normal "warning" now and now it seems to be back at our regular "once a day" frequency. With debug it felt like 5 times a day.

      posted in Support
      P
      patchlings
    • RE: Connector errors, timeout, restart needed.

      @rhessinger Hi, if I run it on the host (or in the actual container) it seems to vary between 25-45, haven't seem anything different between when it behaves bad:

      ubuntu@proget:/opt/provision/additional$ sudo ps huH p 12436 | wc -l
      29
      ubuntu@proget:/opt/provision/additional$ sudo ps huH p 12436 | wc -l
      44
      
      posted in Support
      P
      patchlings
    • RE: Connector errors, timeout, restart needed.

      Performing strace on the mono process gives a lot of:

      [pid 26706] recvfrom(39, "", 8, 0, NULL, NULL) = 0
      [pid 26706] recvfrom(39, "", 8, 0, NULL, NULL) = 0
      [pid 26706] recvfrom(39, "", 8, 0, NULL, NULL) = 0
      [pid 26706] recvfrom(39, "", 8, 0, NULL, NULL) = 0
      [pid 26706] recvfrom(39, "", 8, 0, NULL, NULL) = 0
      [pid 26706] recvfrom(39, "", 8, 0, NULL, NULL) = 0
      [pid 26706] recvfrom(39, "", 8, 0, NULL, NULL) = 0
      

      Then I found this: https://stackoverflow.com/questions/24000534/tracing-socket-file-descriptor-back-to-bound-address

      So the next step will be to see if I can find out which socket that is actually appearing in the strace.

      posted in Support
      P
      patchlings
    • RE: Connector errors, timeout, restart needed.

      One interesting observation is that when the system "freezes" there seems to be no traffic at all (12:30):
      f64b41a1-47e2-4f91-8c61-3fff283a79b3-image.png

      posted in Support
      P
      patchlings
    • RE: Connector errors, timeout, restart needed.

      Now it's seems flaky again:
      nginx status:
      Active connections: 57
      server accepts handled requests
      462208 462208 840656
      Reading: 0 Writing: 30 Waiting: 27

      Main web UI can't be reached at all.
      Ran sudo kill -QUIT $MONO_PID and got myself a huge thread dump with a lot of entries like this (let me know if you want me to supply the entire dump, (it's more then the allowed 32k chars):

      "Thread Pool Worker"  at <unknown> <0xffffffff>
        at (wrapper managed-to-native) object.__icall_wrapper_mono_monitor_enter_v4_internal (object,intptr) [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
        at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext) [0x00035] in <4d8f95b7df544b3fa1a27dda924b4424>:0
        at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult) [0x0014c] in <4d8f95b7df544b3fa1a27dda924b4424>:0
        at System.Net.HttpConnection.OnRead (System.IAsyncResult) [0x00006] in <4d8f95b7df544b3fa1a27dda924b4424>:0
        at System.Net.Sockets.SocketAsyncResult/<>c.<Complete>b__27_0 (object) [0x0000b] in <4d8f95b7df544b3fa1a27dda924b4424>:0
        at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <c453bf849bb84e719dbec0475e99db95>:0
        at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <c453bf849bb84e719dbec0475e99db95>:0
        at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <c453bf849bb84e719dbec0475e99db95>:0
        at (wrapper runtime-invoke) <Module>.runtime_invoke_bool (object,intptr,intptr,intptr) [0x0002a] in <c453bf849bb84e719dbec0475e99db95>:0
      
      posted in Support
      P
      patchlings
    • RE: Connector errors, timeout, restart needed.

      @rhessinger 99% NuGet connector errors.

      posted in Support
      P
      patchlings
    • RE: Connector errors, timeout, restart needed.

      @rhessinger said in Connector errors, timeout, restart needed.:

      Also, how many connectors do you have on your NuGet feed?

      Sorry, I missed this question... We have four connectors (but only uses three of them): connectors.png

      And when I looked that up I noticed that the one we don't use was marked with error, removed it.

      posted in Support
      P
      patchlings