Inedo Community Forums Forums
    • Recent
    • Tags
    • Popular
    • Login

    Welcome to the Inedo Forums! Check out the Forums Guide for help getting started.

    If you are experiencing any issues with the forum software, please visit the Contact Form on our website and let us know!

    Constant timeouts doing NuGet restore

    Scheduled Pinned Locked Moved Support
    23 Posts 4 Posters 66 Views 1 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • B Offline
      brett.polivka
      last edited by

      More info...

      I was able to attach a debugger to the DotNetCore version, and there are approximately 140 threads, all waiting on the same ManualResetEvent. Here's the stack trace:

      System.Private.CoreLib.dll!System.Threading.ManualResetEventSlim.Wait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken) C#
      System.Private.CoreLib.dll!System.Threading.Tasks.Task.SpinThenBlockingWait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken) C#
      System.Private.CoreLib.dll!System.Threading.Tasks.Task.InternalRunSynchronously(System.Threading.Tasks.TaskScheduler scheduler, bool waitForCompletion) C#
      System.Linq.Parallel.dll!System.Linq.Parallel.SpoolingTask.SpoolStopAndGo<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo, int>(System.Linq.Parallel.QueryTaskGroupState groupState, System.Linq.Parallel.PartitionedStream<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo, int> partitions, System.Linq.Parallel.SynchronousChannel<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo>[] channels, System.Threading.Tasks.TaskScheduler taskScheduler) C#
      System.Linq.Parallel.dll!System.Linq.Parallel.DefaultMergeHelper<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo, int>.System.Linq.Parallel.IMergeHelper<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo>.Execute() C#
      System.Linq.Parallel.dll!System.Linq.Parallel.MergeExecutor<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo>.Execute() C#
      System.Linq.Parallel.dll!System.Linq.Parallel.MergeExecutor<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo>.Execute<int>(System.Linq.Parallel.PartitionedStream<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo, int> partitions, bool ignoreOutput, System.Linq.ParallelMergeOptions options, System.Threading.Tasks.TaskScheduler taskScheduler, bool isOrdered, System.Linq.Parallel.CancellationState cancellationState, int queryId) C#
      System.Linq.Parallel.dll!System.Linq.Parallel.PartitionedStreamMerger<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo>.Receive<int>(System.Linq.Parallel.PartitionedStream<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo, int> partitionedStream) C#
      System.Linq.Parallel.dll!System.Linq.Parallel.FirstQueryOperator<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo>.WrapHelper<int>(System.Linq.Parallel.PartitionedStream<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo, int> inputStream, System.Linq.Parallel.IPartitionedStreamRecipient<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo> recipient, System.Linq.Parallel.QuerySettings settings) C#
      System.Linq.Parallel.dll!System.Linq.Parallel.FirstQueryOperator<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo>.WrapPartitionedStream<int>(System.Linq.Parallel.PartitionedStream<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo, int> inputStream, System.Linq.Parallel.IPartitionedStreamRecipient<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo> recipient, bool preferStriping, System.Linq.Parallel.QuerySettings settings) C#
      System.Linq.Parallel.dll!System.Linq.Parallel.UnaryQueryOperator<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo, Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo>.UnaryQueryOperatorResults.ChildResultsRecipient.Receive<int>(System.Linq.Parallel.PartitionedStream<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo, int> inputStream) C#
      System.Linq.Parallel.dll!System.Linq.Parallel.ListQueryResults<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo>.GivePartitionedStream(System.Linq.Parallel.IPartitionedStreamRecipient<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo> recipient) C#
      System.Linq.Parallel.dll!System.Linq.Parallel.UnaryQueryOperator<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo, Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo>.UnaryQueryOperatorResults.GivePartitionedStream(System.Linq.Parallel.IPartitionedStreamRecipient<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo> recipient) C#
      System.Linq.Parallel.dll!System.Linq.Parallel.QueryOperator<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo>.GetOpenedEnumerator(System.Linq.ParallelMergeOptions? mergeOptions, bool suppressOrder, bool forEffect, System.Linq.Parallel.QuerySettings querySettings) C#
      System.Linq.Parallel.dll!System.Linq.Parallel.QueryOpeningEnumerator<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo>.OpenQuery() C#
      System.Linq.Parallel.dll!System.Linq.Parallel.QueryOpeningEnumerator<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo>.MoveNext() C#
      System.Linq.Parallel.dll!System.Linq.ParallelEnumerable.GetOneWithPossibleDefault<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo>(System.Linq.Parallel.QueryOperator<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo> queryOp, bool throwIfTwo, bool defaultIfEmpty) C#
      System.Linq.Parallel.dll!System.Linq.ParallelEnumerable.FirstOrDefault<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo>(System.Linq.ParallelQuery<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo> source, System.Func<Inedo.Web.PageFree.PageFreeHandler.RegexHandlerInfo, bool> predicate) C#
      InedoLib.dll!Inedo.Web.PageFree.PageFreeHandler.GetHttpHandler(Inedo.Web.Handlers.GetHandlerParams args) C#
      InedoLib.dll!Inedo.Web.Handlers.DynamicHttpHandling.GetHandler(System.Web.HttpContext context, string requestType, string url, string pathTranslated) C#
      InedoLib.dll!Inedo.Web.AhWebMiddleware.InvokeAsync(Microsoft.AspNetCore.Http.HttpContext context) C#
      Microsoft.AspNetCore.HostFiltering.dll!Microsoft.AspNetCore.HostFiltering.HostFilteringMiddleware.Invoke(Microsoft.AspNetCore.Http.HttpContext context) C#
      Microsoft.AspNetCore.Hosting.dll!Microsoft.AspNetCore.Hosting.HostingApplication.ProcessRequestAsync(Microsoft.AspNetCore.Hosting.HostingApplication.Context context) C#
      Microsoft.AspNetCore.Server.Kestrel.Core.dll!Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests<Microsoft.AspNetCore.Hosting.HostingApplication.Context>(Microsoft.AspNetCore.Hosting.Server.IHttpApplication<Microsoft.AspNetCore.Hosting.HostingApplication.Context> application) C#
      Microsoft.AspNetCore.Server.Kestrel.Core.dll!Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequestsAsync<Microsoft.AspNetCore.Hosting.HostingApplication.Context>(Microsoft.AspNetCore.Hosting.Server.IHttpApplication<Microsoft.AspNetCore.Hosting.HostingApplication.Context> application) C#
      Microsoft.AspNetCore.Server.Kestrel.Core.dll!Microsoft.AspNetCore.Server.Kestrel.Core.Internal.HttpConnection.ProcessRequestsAsync<Microsoft.AspNetCore.Hosting.HostingApplication.Context>(Microsoft.AspNetCore.Hosting.Server.IHttpApplication<Microsoft.AspNetCore.Hosting.HostingApplication.Context> httpApplication) C#
      Microsoft.AspNetCore.Server.Kestrel.Core.dll!Microsoft.AspNetCore.Server.Kestrel.Core.Internal.HttpConnectionMiddleware<Microsoft.AspNetCore.Hosting.HostingApplication.Context>.OnConnectionAsync(Microsoft.AspNetCore.Connections.ConnectionContext connectionContext) C#
      Microsoft.AspNetCore.Server.Kestrel.Core.dll!Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TransportManager.BindAsync.AnonymousMethod__0(Microsoft.AspNetCore.Connections.ConnectionContext c) C#
      Microsoft.AspNetCore.Server.Kestrel.Core.dll!Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection<Microsoft.AspNetCore.Connections.ConnectionContext>.ExecuteAsync() C#
      Microsoft.AspNetCore.Server.Kestrel.Core.dll!Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.KestrelConnection<System.__Canon>.System.Threading.IThreadPoolWorkItem.Execute() C#
      System.Private.CoreLib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch() C#
      [Native to Managed Transition]

      atrippA 1 Reply Last reply Reply Quote 0
      • atrippA Offline
        atripp inedo-engineer @brett.polivka
        last edited by

        @brett-polivka fantastic, thanks for the additional research. This will make it so much easier to debug from here.

        Do most of the stack traces look like this? I.e. coming from Inedo.Web.PageFree.PageFreeHandler.GetHttpHandler?

        If so, the issue is happening in a totally unexpected place (basically routing a URL). This is just using a relatively simple query with Linq's ParallelEnumerable library methods. Under-the-hood (as the stack trace shows), that seems to leverage operating-system level mutex support... which is likely why it's problematic only on your specific hardware/environment.

        We saw a ton of Mono-related bugs early-on with this (some even would SEGFAULT on certain hosts), largely because we tend to find the "corner-cases" with optimizations, like working-around Activator.CreateInstance slowness using using System.Reflection.Emit directly.

        Anyways, I'm going to have to involve our engineering team for ideas, because at this point we have to guess how to work-around this.

        1 Reply Last reply Reply Quote 0
        • B Offline
          brett.polivka
          last edited by

          Glad to help.

          Yes, almost every thread shows that exact same stack trace.

          If there's any additional testing or information I can provide, please let me know.

          1 Reply Last reply Reply Quote 0
          • B Offline
            brett.polivka
            last edited by

            One more piece of of info... I was able run the same test on Windows (I extracted the apps from the Docker image and copied them to a Windows machine), and it exhibits exactly the same behavior.

            1 Reply Last reply Reply Quote 0
            • B Offline
              brett.polivka
              last edited by

              If I can venture a guess as to what's happening, I think that the ThreadPool is being completely saturated.

              Kestrel will use the ThreadPool to service a request. That request is then using a parallel query, which will also use the ThreadPool. At some point, every thread in the pool is blocked, waiting for a task to complete, but the task can't complete because it needs to be scheduled onto the ThreadPool, but every thread in the pool is blocked.

              I'd be curious to see what happens if the ParallelEnumerable query is replaced with a standard Enumerable query.

              atrippA 1 Reply Last reply Reply Quote 0
              • atrippA Offline
                atripp inedo-engineer @brett.polivka
                last edited by

                @brett-polivka thanks again for all the help and insights!

                I talked this over with the engineering team, and we just decided to remove the ParallelEnumerable query.

                Would you be able to try a pre-release container?

                inedo/proget:5.3.23-ci.4

                1 Reply Last reply Reply Quote 0
                • B Offline
                  brett.polivka
                  last edited by

                  I'm still seeing the same problem and the stack traces. Can you confirm that the 5.3.23-ci.4 release is the right version, because it still contains the parallel query.

                  1 Reply Last reply Reply Quote 0
                  • rhessingerR Offline
                    rhessinger inedo-engineer
                    last edited by

                    Hi @brett-polivka,

                    It looks like there was a mix-up in one of our branches. I just pushed an updated image with the ParallelEnumerable removed. Can you please give the proget.inedo.com/productimages/inedo/proget:5.3.23-ci.5 image a try?

                    Thanks,
                    Rich

                    Products Engineer, Inedo

                    1 Reply Last reply Reply Quote 0
                    • B Offline
                      brett.polivka
                      last edited by

                      Hi Rich,

                      This seems to have done the trick! I've been running constant restores against it for an hour with no timeouts.

                      Thanks!!

                      1 Reply Last reply Reply Quote 0
                      • B Offline
                        brett.polivka
                        last edited by

                        Thanks to @atripp as well. Appreciate your patience dealing with this (and me).

                        atrippA 1 Reply Last reply Reply Quote 0
                        • atrippA Offline
                          atripp inedo-engineer @brett.polivka
                          last edited by

                          Whoops, my bad!

                          @brett-polivka thanks so much for the help, this was a baffling .NET5 issue, but so glad we could identify and fix it. This will definitely go in the maintence release as well, of course.

                          N 1 Reply Last reply Reply Quote 0
                          • N Offline
                            nuno.guerreiro.rosa_9280 @atripp
                            last edited by

                            @atripp This is probably related with the same issue i kept mentioning which you were saying it was possibly due to an overloaded network stack or other type of configurations. I still have the problem even after setting up a completely new server so it also shows that any timeout is related with proget application code. Looking forward to try out this fix. This apparently was happening to other users as well. :)

                            atrippA 1 Reply Last reply Reply Quote 1
                            • atrippA Offline
                              atripp inedo-engineer @nuno.guerreiro.rosa_9280
                              last edited by

                              @nuno-guerreiro-rosa_9280 very glad we could fix it! It's definitely a .NET5 bug, and in a totally unexpected place

                              1 Reply Last reply Reply Quote 0
                              • N Offline
                                nuno.guerreiro.rosa_9280
                                last edited by

                                I also confirm I didn't get any more timeouts on the latest version. Very good!

                                1 Reply Last reply Reply Quote 0

                                Hello! It looks like you're interested in this conversation, but you don't have an account yet.

                                Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.

                                With your input, this post could be even better 💗

                                Register Login
                                • 1
                                • 2
                                • 1 / 2
                                • First post
                                  Last post
                                Inedo Website Home • Support Home • Code of Conduct • Forums Guide • Documentation