Connector errors, timeout, restart needed.

rhessinger

Thanks for all this information. Would you be able to get me an open thread count for when everything is running fine and when things are not? You should be able to find it by running sudo ps huH p <PID_OF_MONO_PROCESS> | wc -l. Also, are you running these commands inside the docker image? Or on the host?

Thanks,
Rich

patchlings

@rhessinger Hi, if I run it on the host (or in the actual container) it seems to vary between 25-45, haven't seem anything different between when it behaves bad:

ubuntu@proget:/opt/provision/additional$ sudo ps huH p 12436 | wc -l
29
ubuntu@proget:/opt/provision/additional$ sudo ps huH p 12436 | wc -l
44

rhessinger

Hi @patchlings,

Thank you for checking this for me. So the good thing is that this is definitely not a threading issue. Can you give me a stub status when it's working during normal use and one after it starts behaving poorly?

Thanks,
Rich

patchlings

@rhessinger I will do that.

One note, it seems that running on debug logging level makes it a little more unstable. We are back to normal "warning" now and now it seems to be back at our regular "once a day" frequency. With debug it felt like 5 times a day.

rhessinger

Hi @patchlings,

Thanks! I'm not surprised that debug exaccerbates the issue, but it's definitely good to know.

Thanks,
Rich

patchlings

Now it misbehaves...

Active connections: 6 
server accepts handled requests
 115659 115659 179524 
Reading: 0 Writing: 1 Waiting: 5

Accessing feeds/default will show a "There was an error with a connector: The operation has timed out." All other feeds work as expected.

ubuntu@proget:~$ sudo ps huH p 18819 |wc -l
33

In the admin interface:

Request URL: https://nuget.org/api/v2/Search()?$top=25&$filter=IsAbsoluteLatestVersion&searchTerm=%27%27&targetFramework=%27%27&includePrerelease=true&semVerLevel=2.0.0

System.Net.WebException: The operation has timed out.
at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task`1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func`1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.HttpWebRequest.EndGetResponse (System.IAsyncResult asyncResult) [0x00020] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Threading.Tasks.TaskFactory`1[TResult].FromAsyncCoreLogic (System.IAsyncResult iar, System.Func`2[T,TResult] endFunction, System.Action`1[T] endAction, System.Threading.Tasks.Task`1[TResult] promise, System.Boolean requiresSynchronization) [0x0000f] in <c453bf849bb84e719dbec0475e99db95>:0
--- End of stack trace from previous location where exception was thrown ---

at Inedo.ProGet.Feeds.StrongConnector.GetDirectResponseAsync (System.String url) [0x000eb] in <dacb7de90ce24964b2da21b67ac5e1d1>:0
at Inedo.ProGet.Feeds.StrongConnector.GetCachedResponseAsync (System.String url) [0x000a8] in <dacb7de90ce24964b2da21b67ac5e1d1>:0
at Inedo.ProGet.Feeds.StrongConnector.GetCachedTextReaderAsync (System.String url) [0x00075] in <dacb7de90ce24964b2da21b67ac5e1d1>:0
at Inedo.ProGet.Feeds.NuGet.NuGetFeedClient.PerformRequestWithAbsoluteUrlAsync (System.String fullUrl) [0x000d9] in <dacb7de90ce24964b2da21b67ac5e1d1>:0

strace doesn't show anything obvious.

Restart docker container and now the /feed/default shows no problem. nginx status:

Active connections: 6 
server accepts handled requests
 115829 115829 180229 
Reading: 0 Writing: 1 Waiting: 5

rhessinger

Hi @patchlings,

Thanks for all the data. I really appreciate your help with this! I'm currently reviewing our code to see if I can find a correlation with this.

Thanks,
Rich

rhessinger

Hi @patchlings,

We believe this error is due to a regression/bug in the Mono, which is related to change to their HttpWebRequest stack. In the future, after Microsoft releases it, we plan to move to .NET 5 for a more stable platform.

The best resolution, for now, is to move to Windows/IIS; it's the most stable. Is this possible?

If not... we don't know when Mono will fix the bug, or if it's possible for us to work-around, but we think it will get resolved in the future. Would you be able to make ProGet restart once per day? I think this would help with the memory leak.

Thanks,
Rich

patchlings

Hi @rhessinger

Ok, the issue you found sounds interesting, but it seems to be more connected to the other problem we have: https://forums.inedo.com/topic/2887/proget-in-docker-linux-hanging-after-using-all-memory/6

And we are currently already restarting the docker container once a day to make sure that we don't run out of memory.

And looking into the memory graphs it looks like we experience the connection issue even if the memory consumption looks ok (Green bars = scheduled restarts, red bars = connector problems, yellow graph = available memory, green graph = free memory):

-- Magnus

rhessinger

Hi @patchlings ,

That is the heart of the Mono bug that I referenced. Currently Mono has a memory leak around HTTP requests which will cause our connectors to hang when the memory usage is too high. They have reduced the leaking substantially starting in Mono 5.16, but there is still a leak. If you search around on this error, you will find similar questions with almost identical errors still in 2020. Unfortunately, we are in the hands of the Mono team until we switch off Mono and migrate to .NET 5.

Thanks,
Rich

patchlings

@rhessinger Ok, then I understand. We will look into our options then. Depending on Mono fixing this seems like a bad solution as it seems to have appeared for quite some time.

Thanks for all your help!

What is the timeline of your Mono to .NET 5 conversion?

rhessinger

Hi @patchlings,

No problem! We are still investigating some workarounds, but we don't have anything specific yet.

We don't currently have a timeline for the .NET 5 switch. We have attempted a .NET Core transition in early 2019, but we ran into too many instability issues and scrapped it. We will wait for .NET 5 to be officially released (Microsoft says November 2020) before we attempt the switch, but it will all come down to the stability of ProGet on .NET 5.

Thanks,
Rich