Inedo Community Forums Forums
    • Recent
    • Tags
    • Popular
    • Login

    Welcome to the Inedo Forums! Check out the Forums Guide for help getting started.

    If you are experiencing any issues with the forum software, please visit the Contact Form on our website and let us know!

    Constant timeouts doing NuGet restore

    Scheduled Pinned Locked Moved Support
    23 Posts 4 Posters 66 Views 1 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • B Offline
      brett.polivka
      last edited by

      As a test I reverted to the 5.3.19 Mono version, and it is extremely fast and completes the entire restore within seconds.

      I then flipped back to the 5.3.19 .NET Core version, and it is so slow that the restore will not complete.

      Nothing else changed between these two runs other than Mono vs. .NET Core.

      1 Reply Last reply Reply Quote 0
      • atrippA Offline
        atripp inedo-engineer
        last edited by

        Thanks for the update @brett-polivka

        The ProGet code (application layer) is effectively identical between those images; the main differences is are the underlying container image and .NET5 vs Mono. We had some initial bugs with .NET5, but is the first we've heard of Mono vs DotNetCore problems of this nature out of a lot of users.

        Unfortunately, we really don't have any means to diagnose this further at this point. We're not even sure what code we could put in place to do network / operating-system level debugging. And worse, we can't reproduce this in our containers, even on Azure host.

        We're open to ideas here, but I think your best bet would be go to as basic of an installation as possible; just a simple container with a simple database, and then add the pieces back until you figure out which thing is causing problems.

        1 Reply Last reply Reply Quote 0
        • B Offline
          brett.polivka
          last edited by

          While I wouldn't think it would be enough to cause the type of performance issues I'm seeing, I've noticed that the DotNetCore version logs significantly more data to the console than the mono version.

          Is there a way to configure the console log level?

          1 Reply Last reply Reply Quote 0
          • B Offline
            brett.polivka
            last edited by

            More info...

            I was able to attach a debugger to the DotNetCore version, and there are approximately 140 threads, all waiting on the same ManualResetEvent. Here's the stack trace:

            System.Private.CoreLib.dll!System.Threading.ManualResetEventSlim.Wait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken) C#
            System.Private.CoreLib.dll!System.Threading.Tasks.Task.SpinThenBlockingWait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken) C#
            System.Private.CoreLib.dll!System.Threading.Tasks.Task.InternalRunSynchronously(System.Threading.Tasks.TaskScheduler scheduler, bool waitForCompletion) C#
            System.Linq.Parallel.dll!System.Linq.Parallel.SpoolingTask.SpoolStopAndGo<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo, int>(System.Linq.Parallel.QueryTaskGroupState groupState, System.Linq.Parallel.PartitionedStream<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo, int> partitions, System.Linq.Parallel.SynchronousChannel<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo>[] channels, System.Threading.Tasks.TaskScheduler taskScheduler) C#
            System.Linq.Parallel.dll!System.Linq.Parallel.DefaultMergeHelper<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo, int>.System.Linq.Parallel.IMergeHelper<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo>.Execute() C#
            System.Linq.Parallel.dll!System.Linq.Parallel.MergeExecutor<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo>.Execute() C#
            System.Linq.Parallel.dll!System.Linq.Parallel.MergeExecutor<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo>.Execute<int>(System.Linq.Parallel.PartitionedStream<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo, int> partitions, bool ignoreOutput, System.Linq.ParallelMergeOptions options, System.Threading.Tasks.TaskScheduler taskScheduler, bool isOrdered, System.Linq.Parallel.CancellationState cancellationState, int queryId) C#
            System.Linq.Parallel.dll!System.Linq.Parallel.PartitionedStreamMerger<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo>.Receive<int>(System.Linq.Parallel.PartitionedStream<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo, int> partitionedStream) C#
            System.Linq.Parallel.dll!System.Linq.Parallel.FirstQueryOperator<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo>.WrapHelper<int>(System.Linq.Parallel.PartitionedStream<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo, int> inputStream, System.Linq.Parallel.IPartitionedStreamRecipient<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo> recipient, System.Linq.Parallel.QuerySettings settings) C#
            System.Linq.Parallel.dll!System.Linq.Parallel.FirstQueryOperator<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo>.WrapPartitionedStream<int>(System.Linq.Parallel.PartitionedStream<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo, int> inputStream, System.Linq.Parallel.IPartitionedStreamRecipient<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo> recipient, bool preferStriping, System.Linq.Parallel.QuerySettings settings) C#
            System.Linq.Parallel.dll!System.Linq.Parallel.UnaryQueryOperator<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo, Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo>.UnaryQueryOperatorResults.ChildResultsRecipient.Receive<int>(System.Linq.Parallel.PartitionedStream<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo, int> inputStream) C#
            System.Linq.Parallel.dll!System.Linq.Parallel.ListQueryResults<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo>.GivePartitionedStream(System.Linq.Parallel.IPartitionedStreamRecipient<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo> recipient) C#
            System.Linq.Parallel.dll!System.Linq.Parallel.UnaryQueryOperator<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo, Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo>.UnaryQueryOperatorResults.GivePartitionedStream(System.Linq.Parallel.IPartitionedStreamRecipient<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo> recipient) C#
            System.Linq.Parallel.dll!System.Linq.Parallel.QueryOperator<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo>.GetOpenedEnumerator(System.Linq.ParallelMergeOptions? mergeOptions, bool suppressOrder, bool forEffect, System.Linq.Parallel.QuerySettings querySettings) C#
            System.Linq.Parallel.dll!System.Linq.Parallel.QueryOpeningEnumerator<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo>.OpenQuery() C#
            System.Linq.Parallel.dll!System.Linq.Parallel.QueryOpeningEnumerator<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo>.MoveNext() C#
            System.Linq.Parallel.dll!System.Linq.ParallelEnumerable.GetOneWithPossibleDefault<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo>(System.Linq.Parallel.QueryOperator<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo> queryOp, bool throwIfTwo, bool defaultIfEmpty) C#
            System.Linq.Parallel.dll!System.Linq.ParallelEnumerable.FirstOrDefault<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo>(System.Linq.ParallelQuery<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo> source, System.Func<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo, bool> predicate) C#
            InedoLib.dll!Inedo.Web.PageFree.PageFreeHandler.GetHttpHandler(Inedo.Web.Handlers.GetHandlerParams args) C#
            InedoLib.dll!Inedo.Web.Handlers.DynamicHttpHandling.GetHandler(System.Web.HttpContext context, string requestType, string url, string pathTranslated) C#
            InedoLib.dll!Inedo.Web.AhWebMiddleware.InvokeAsync(Microsoft.AspNetCore.Http.HttpContext context) C#
            Microsoft.AspNetCore.HostFiltering.dll!Microsoft.AspNetCore.HostFiltering.HostFilteringMiddleware.Invoke(Microsoft.AspNetCore.Http.HttpContext context) C#
            Microsoft.AspNetCore.Hosting.dll!Microsoft.AspNetCore.Hosting.HostingApplication.ProcessRequestAsync(Microsoft.AspNetCore.Hosting.HostingApplication.Context context) C#
            Microsoft.AspNetCore.Server.Kestrel.Core.dll!Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests<Microsoft.AspNetCore.Hosting.HostingApplication.Context>(Microsoft.AspNetCore.Hosting.Server.IHttpApplication<Microsoft.AspNetCore.Hosting.HostingApplication.Context> application) C#
            Microsoft.AspNetCore.Server.Kestrel.Core.dll!Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequestsAsync<Microsoft.AspNetCore.Hosting.HostingApplication.Context>(Microsoft.AspNetCore.Hosting.Server.IHttpApplication<Microsoft.AspNetCore.Hosting.HostingApplication.Context> application) C#
            Microsoft.AspNetCore.Server.Kestrel.Core.dll!Microsoft.AspNetCore.Server.Kestrel.Core.Internal.HttpConnection.ProcessRequestsAsync<Microsoft.AspNetCore.Hosting.HostingApplication.Context>(Microsoft.AspNetCore.Hosting.Server.IHttpApplication<Microsoft.AspNetCore.Hosting.HostingApplication.Context> httpApplication) C#
            Microsoft.AspNetCore.Server.Kestrel.Core.dll!Microsoft.AspNetCore.Server.Kestrel.Core.Internal.HttpConnectionMiddleware<Microsoft.AspNetCore.Hosting.HostingApplication.Context>.OnConnectionAsync(Microsoft.AspNetCore.Connections.ConnectionContext connectionContext) C#
            Microsoft.AspNetCore.Server.Kestrel.Core.dll!Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TransportManager.BindAsync.AnonymousMethod__0(Microsoft.AspNetCore.Connections.ConnectionContext c) C#
            Microsoft.AspNetCore.Server.Kestrel.Core.dll!Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection<Microsoft.AspNetCore.Connections.ConnectionContext>.ExecuteAsync() C#
            Microsoft.AspNetCore.Server.Kestrel.Core.dll!Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection<System.__Canon>.System.Threading.IThreadPoolWorkItem.Execute() C#
            System.Private.CoreLib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch() C#
            [Native to Managed Transition]

            atrippA 1 Reply Last reply Reply Quote 0
            • atrippA Offline
              atripp inedo-engineer @brett.polivka
              last edited by

              @brett-polivka fantastic, thanks for the additional research. This will make it so much easier to debug from here.

              Do most of the stack traces look like this? I.e. coming from Inedo.Web.PageFree.PageFreeHandler.GetHttpHandler?

              If so, the issue is happening in a totally unexpected place (basically routing a URL). This is just using a relatively simple query with Linq's ParallelEnumerable library methods. Under-the-hood (as the stack trace shows), that seems to leverage operating-system level mutex support... which is likely why it's problematic only on your specific hardware/environment.

              We saw a ton of Mono-related bugs early-on with this (some even would SEGFAULT on certain hosts), largely because we tend to find the "corner-cases" with optimizations, like working-around Activator.CreateInstance slowness using using System.Reflection.Emit directly.

              Anyways, I'm going to have to involve our engineering team for ideas, because at this point we have to guess how to work-around this.

              1 Reply Last reply Reply Quote 0
              • B Offline
                brett.polivka
                last edited by

                Glad to help.

                Yes, almost every thread shows that exact same stack trace.

                If there's any additional testing or information I can provide, please let me know.

                1 Reply Last reply Reply Quote 0
                • B Offline
                  brett.polivka
                  last edited by

                  One more piece of of info... I was able run the same test on Windows (I extracted the apps from the Docker image and copied them to a Windows machine), and it exhibits exactly the same behavior.

                  1 Reply Last reply Reply Quote 0
                  • B Offline
                    brett.polivka
                    last edited by

                    If I can venture a guess as to what's happening, I think that the ThreadPool is being completely saturated.

                    Kestrel will use the ThreadPool to service a request. That request is then using a parallel query, which will also use the ThreadPool. At some point, every thread in the pool is blocked, waiting for a task to complete, but the task can't complete because it needs to be scheduled onto the ThreadPool, but every thread in the pool is blocked.

                    I'd be curious to see what happens if the ParallelEnumerable query is replaced with a standard Enumerable query.

                    atrippA 1 Reply Last reply Reply Quote 0
                    • atrippA Offline
                      atripp inedo-engineer @brett.polivka
                      last edited by

                      @brett-polivka thanks again for all the help and insights!

                      I talked this over with the engineering team, and we just decided to remove the ParallelEnumerable query.

                      Would you be able to try a pre-release container?

                      inedo/proget:5.3.23-ci.4

                      1 Reply Last reply Reply Quote 0
                      • B Offline
                        brett.polivka
                        last edited by

                        I'm still seeing the same problem and the stack traces. Can you confirm that the 5.3.23-ci.4 release is the right version, because it still contains the parallel query.

                        1 Reply Last reply Reply Quote 0
                        • rhessingerR Offline
                          rhessinger inedo-engineer
                          last edited by

                          Hi @brett-polivka,

                          It looks like there was a mix-up in one of our branches. I just pushed an updated image with the ParallelEnumerable removed. Can you please give the proget.inedo.com/productimages/inedo/proget:5.3.23-ci.5 image a try?

                          Thanks,
                          Rich

                          Products Engineer, Inedo

                          1 Reply Last reply Reply Quote 0
                          • B Offline
                            brett.polivka
                            last edited by

                            Hi Rich,

                            This seems to have done the trick! I've been running constant restores against it for an hour with no timeouts.

                            Thanks!!

                            1 Reply Last reply Reply Quote 0
                            • B Offline
                              brett.polivka
                              last edited by

                              Thanks to @atripp as well. Appreciate your patience dealing with this (and me).

                              atrippA 1 Reply Last reply Reply Quote 0
                              • atrippA Offline
                                atripp inedo-engineer @brett.polivka
                                last edited by

                                Whoops, my bad!

                                @brett-polivka thanks so much for the help, this was a baffling .NET5 issue, but so glad we could identify and fix it. This will definitely go in the maintence release as well, of course.

                                N 1 Reply Last reply Reply Quote 0
                                • N Offline
                                  nuno.guerreiro.rosa_9280 @atripp
                                  last edited by

                                  @atripp This is probably related with the same issue i kept mentioning which you were saying it was possibly due to an overloaded network stack or other type of configurations. I still have the problem even after setting up a completely new server so it also shows that any timeout is related with proget application code. Looking forward to try out this fix. This apparently was happening to other users as well. :)

                                  atrippA 1 Reply Last reply Reply Quote 1
                                  • atrippA Offline
                                    atripp inedo-engineer @nuno.guerreiro.rosa_9280
                                    last edited by

                                    @nuno-guerreiro-rosa_9280 very glad we could fix it! It's definitely a .NET5 bug, and in a totally unexpected place

                                    1 Reply Last reply Reply Quote 0
                                    • N Offline
                                      nuno.guerreiro.rosa_9280
                                      last edited by

                                      I also confirm I didn't get any more timeouts on the latest version. Very good!

                                      1 Reply Last reply Reply Quote 0

                                      Hello! It looks like you're interested in this conversation, but you don't have an account yet.

                                      Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.

                                      With your input, this post could be even better 💗

                                      Register Login
                                      • 1
                                      • 2
                                      • 1 / 2
                                      • First post
                                        Last post
                                      Inedo Website Home • Support Home • Code of Conduct • Forums Guide • Documentation