Inedo Community Forums Forums
    • Recent
    • Tags
    • Popular
    • Login

    Welcome to the Inedo Forums! Check out the Forums Guide for help getting started.

    If you are experiencing any issues with the forum software, please visit the Contact Form on our website and let us know!

    Nuget Feed Connection timeout

    Scheduled Pinned Locked Moved Support
    22 Posts 4 Posters 85 Views 1 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • J Offline
      jyip_5228
      last edited by

      I am using nuget v3 api but for some reason getting connection timeout on v2.
      cfa6ed02-5ffd-41fd-a76f-76f46fc00b81-image.png
      b4b2c556-5a95-49c7-9ef5-3c2f16e5aa07-image.png

      Request URL: https://www.nuget.org/api/v2/FindPackagesById()?id=%27microsoft.extensions.dependencyinjection.abstractions%27&semVerLevel=2.0.0

System.Net.WebException: The operation has timed out.
at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task`1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func`1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Net.HttpWebRequest.EndGetResponse (System.IAsyncResult asyncResult) [0x00020] in <4d8f95b7df544b3fa1a27dda924b4424>:0
at System.Threading.Tasks.TaskFactory`1[TResult].FromAsyncCoreLogic (System.IAsyncResult iar, System.Func`2[T,TResult] endFunction, System.Action`1[T] endAction, System.Threading.Tasks.Task`1[TResult] promise, System.Boolean requiresSynchronization) [0x0000f] in <c453bf849bb84e719dbec0475e99db95>:0
--- End of stack trace from previous location where exception was thrown ---

at Inedo.ProGet.Feeds.Connector.GetDirectResponseAsync (System.String url) [0x000eb] in <5675ed1ff2ff49938725a260ae4389ad>:0
at Inedo.ProGet.Feeds.Connector.GetCachedResponseAsync (System.String url) [0x000a8] in <5675ed1ff2ff49938725a260ae4389ad>:0
at Inedo.ProGet.Feeds.Connector.GetCachedTextReaderAsync (System.String url) [0x00075] in <5675ed1ff2ff49938725a260ae4389ad>:0
at Inedo.ProGet.Feeds.NuGet.V2.NuGetFeedClient.PerformRequestWithAbsoluteUrlAsync (System.String fullUrl) [0x000d9] in <5675ed1ff2ff49938725a260ae4389ad>:0
      1 Reply Last reply Reply Quote 0
      • atrippA Offline
        atripp inedo-engineer
        last edited by

        At first, I wouldn't worry about connector timeout errors to nuget.org unless it's frequent. They happen due to service or network outages (your end or theirs), nothing to worry about too much.

        But otherwise it's hard to say; are you totally sure that the error is coming from that connector? Maybe you have multiple connectors to nuget.org?

        Otherwise, nuget,.org's JSON-LD API (v3) sometimes points to the nuget.org ODATA-based URLS (v2), though you'd have to follow the URLs returned by the v3 API to see that.

        1 Reply Last reply Reply Quote 0
        • J Offline
          jyip_5228
          last edited by

          I had 2 nuget connectors originally, v2 and v3, but I deleted the v2 just to see if this error would stop. But it still happened. Works for me if its nothing to be concerned about.

          thank you.

          1 Reply Last reply Reply Quote 0
          • J Offline
            jyip_5228
            last edited by

            actually I would like to add that Proget is also reaching out to nuget.org to search for internal packages as well and gets a timeout. Is it suppose to reach out for something that it should have found locally?
            I wonder if these additional calls to nuget.org contributes to to this exception that I have also been getting.

            Details: System.InvalidOperationException: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
            1 Reply Last reply Reply Quote 0
            • rhessingerR Offline
              rhessinger inedo-engineer
              last edited by

              Hi @jyip_5228,

              The last exception you are seeing is currently a known issue that we are currently working through. We have noticed this tends to affect the docker based ProGet installations the most. In most cases, increasing the max connection pool limit in the SQL connection string seems to fix the issue. Depending on your SQL Server setup, you may need to increase the allowed connections on SQL Server as well. Can you please try to increase the max connection pool in your SQL connection string? Also, can you please tell us what version and edition of SQL Server you are currently running?

              Thanks,
              Rich

              Products Engineer, Inedo

              1 Reply Last reply Reply Quote 0
              • J Offline
                jyip_5228
                last edited by

                i am running MSSQL 2017 in docker, will that require increasing the max allowed connections on the SQL server as well? I will increase the max pool size to 300 and see how that goes.

                1 Reply Last reply Reply Quote 0
                • rhessingerR Offline
                  rhessinger inedo-engineer
                  last edited by

                  Hi @jyip_5228,

                  The default setting for SQL Server 2017 is unlimited. As long as you didnt set one on your SQL Server, then you should be good.

                  Thanks,
                  Rich

                  Products Engineer, Inedo

                  1 Reply Last reply Reply Quote 0
                  • J Offline
                    jyip_5228
                    last edited by

                    i set it to 300 in the connection string still seeing max pool, gonna increase it to 500 and see how that goes.

                    1 Reply Last reply Reply Quote 0
                    • J Offline
                      jyip_5228
                      last edited by

                      had to up it to 1000 before i stopped seeing the issue

                      1 Reply Last reply Reply Quote 0
                      • rhessingerR Offline
                        rhessinger inedo-engineer
                        last edited by

                        Hi @jyip_5228,

                        Thanks for giving me the updated counts. Glad this workaround is working for you. We are still working on the issue internally. Do you happen to notice any other errors in the logs right before the timeout happens?

                        Thanks,
                        Rich

                        Products Engineer, Inedo

                        1 Reply Last reply Reply Quote 0
                        • J Offline
                          jyip_5228
                          last edited by

                          Actually seems like 1000 is not enough, i just hit the error again upon compiling a bigger project. These are the two errors that are grouped together usually.

                           An error occurred processing a GET request to http://xxxxx.nupkg: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.

                          An error occurred in the web application: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
                          1 Reply Last reply Reply Quote 0
                          • rhessingerR Offline
                            rhessinger inedo-engineer
                            last edited by

                            Hi @jyip_5228,

                            Thanks for the update. I'm currently implementing a potential fix to the connection stuff. I'll let you know when I have something ready.

                            Thanks,
                            Rich

                            Products Engineer, Inedo

                            1 Reply Last reply Reply Quote 0
                            • rhessingerR Offline
                              rhessinger inedo-engineer
                              last edited by

                              Hi @jyip_5228,

                              Just an update! We will be shipping a potential fix in PG-1783, which adds a new checkbox in advanced settings (unchecked by default):

                              ☑ Close Database Connections Early
                              EXPERIMENTAL (for debugging/testing only) - As of ProGet 5.3, database connections are left open during the lifecycle of a NuGet API request as a means to reduce overhead; however, this may be causing ProGet to run out of available connections to SQL Server. Set this value true to open/close database connections as needed on NuGet feeds.

                              We'll update when this is shipped --- but if we can get some folks to verify that this works better (we can't repro, at all 😦 ), then we will likely make it the default. Hopefully this will do it. Seems better than raising connection pool limits 🤔

                              If so, then the savings in connection open/close overhead don't seem to make it particularly worthwhile. This "keep open" technique made a ton of differences elsewhere in our software, but since a NuGet API requests may yield a ton of other network requests (via connectors) and block, the pool may be getting drained too quickly...

                              just a theory, as I mentioned. Anyway hope this helps, stay tuned!!

                              Thanks!

                              Products Engineer, Inedo

                              1 Reply Last reply Reply Quote 0
                              • J Offline
                                jyip_5228
                                last edited by

                                Great! hoping this will fix the issue.

                                1 Reply Last reply Reply Quote 0
                                • rhessingerR Offline
                                  rhessinger inedo-engineer
                                  last edited by

                                  Hi @jyip_5228 ,

                                  We just released ProGet 5.3.8, which includes the Close Database Connections Early option in the advanced settings. Could you please upgrade and give this a try?

                                  Thanks,
                                  Rich

                                  Products Engineer, Inedo

                                  1 Reply Last reply Reply Quote 0
                                  • J Offline
                                    jyip_5228
                                    last edited by

                                    hi, it looks good so far, will see if it helps other issues as well.

                                    1 Reply Last reply Reply Quote 0
                                    • J Offline
                                      jyip_5228
                                      last edited by

                                      Hi I'm running 5.3.10 and getting timeouts again / max pool reached, is the database issue occurring again?

                                      An error occurred in the web application: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
 Details: URL: http://xxx/nuget/ks-nuget/v3/flatcontainer/xxx/xxx/xxxxx.nupkg
Referrer: (not set)
User: (unknown)
User Agent: NuGet Desktop MSBuild Task/5.5.0 (Microsoft Windows NT 10.0.18363.0)
Stack trace: at System.Data.ProviderBase.DbConnectionFactory.TryGetConnection (System.Data.Common.DbConnection owningConnection, System.Threading.Tasks.TaskCompletionSource`1[TResult] retry, System.Data.Common.DbConnectionOptions userOptions, System.Data.ProviderBase.DbConnectionInternal oldConnection, System.Data.ProviderBase.DbConnectionInternal& connection) [0x0024d] in <0864334e7e474248b37afca9b637daa9>:0
at System.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal (System.Data.Common.DbConnection outerConnection, System.Data.ProviderBase.DbConnectionFactory connectionFactory, System.Threading.Tasks.TaskCompletionSource`1[TResult] retry, System.Data.Common.DbConnectionOptions userOptions) [0x00036] in <0864334e7e474248b37afca9b637daa9>:0
at System.Data.ProviderBase.DbConnectionClosed.TryOpenConnection (System.Data.Common.DbConnection outerConnection, System.Data.ProviderBase.DbConnectionFactory connectionFactory, System.Threading.Tasks.TaskCompletionSource`1[TResult] retry, System.Data.Common.DbConnectionOptions userOptions) [0x00000] in <0864334e7e474248b37afca9b637daa9>:0
at System.Data.SqlClient.SqlConnection.TryOpen (System.Threading.Tasks.TaskCompletionSource`1[TResult] retry) [0x0005d] in <0864334e7e474248b37afca9b637daa9>:0
at System.Data.SqlClient.SqlConnection.Open () [0x0003b] in <0864334e7e474248b37afca9b637daa9>:0
at Inedo.Data.SqlServerDatabaseContext.CreateConnection () [0x0006c] in <8a0468ee94a94ef1b2bba97871f29990>:0
at Inedo.Data.DatabaseContext.ExecuteInternal (System.String storedProcName, Inedo.Data.GenericDbParameter[] parameters) [0x00064] in <8a0468ee94a94ef1b2bba97871f29990>:0
at Inedo.Data.SqlServerDatabaseContext.ExecuteInternal (System.String storedProcName, Inedo.Data.GenericDbParameter[] parameters) [0x00005] in <8a0468ee94a94ef1b2bba97871f29990>:0
at Inedo.Data.DatabaseContext+<>c__DisplayClass29_0`1[TRow].<EnumerateTable>b__0 () [0x00000] in <8a0468ee94a94ef1b2bba97871f29990>:0
at Inedo.Data.StrongDataReader+<Read>d__11`1[TRow].MoveNext () [0x0003e] in <8a0468ee94a94ef1b2bba97871f29990>:0
at System.Linq.Enumerable.TryGetFirst[TSource] (System.Collections.Generic.IEnumerable`1[T] source, System.Boolean& found) [0x00045] in <22384ee444974b39bb55b725de39c721>:0
at System.Linq.Enumerable.FirstOrDefault[TSource] (System.Collections.Generic.IEnumerable`1[T] source) [0x00000] in <22384ee444974b39bb55b725de39c721>:0
at Inedo.ProGet.Data.DB+Context.Feeds_GetFeed (System.Nullable`1[T] Feed_Id, System.String Feed_Name) [0x0003d] in <888866fba54f44d68ecb60a2adf95719>:0
at Inedo.ProGet.Feeds.Feed.GetFeed (System.String feedName, Inedo.ProGet.Data.DB+Context externalDbContext) [0x0000b] in <888866fba54f44d68ecb60a2adf95719>:0
at Inedo.ProGet.WebApplication.FeedEndpoints.FeedEndpointHandler.GetHttpHandler (Inedo.Web.Handlers.GetHandlerParams args) [0x0017e] in <d560b10d729641519d596b0266d076a4>:0
at Inedo.Web.Handlers.DynamicHttpHandling.GetHandler (System.Web.HttpContext context, System.String requestType, System.String url, System.String pathTranslated) [0x0003b] in <8a0468ee94a94ef1b2bba97871f29990>:0
at Inedo.Web.InedoHttpModule.MapHandlerAndBeginRequestAsync (System.Web.HttpApplication app) [0x00029] in <8a0468ee94a94ef1b2bba97871f29990>:0
at Inedo.Web.InedoHttpModule.HandleBeginRequestInternalAsync (System.Web.HttpApplication app) [0x0000f] in <8a0468ee94a94ef1b2bba97871f29990>:0
at Inedo.Web.InedoHttpModule.ProcessBegin (System.Object sender, System.EventArgs e, System.AsyncCallback cb, System.Object extraData) [0x0000d] in <8a0468ee94a94ef1b2bba97871f29990>:0
at System.Web.HttpApplication+<RunHooks>d__217.MoveNext () [0x000a4] in <b4f0b153c02f4f0588d3f7549d75281b>:0
at System.Web.HttpApplication+<Pipeline>d__225.MoveNext () [0x0012c] in <b4f0b153c02f4f0588d3f7549d75281b>:0
at System.Web.HttpApplication.Tick () [0x00000] in <b4f0b153c02f4f0588d3f7549d75281b>:0
                                      1 Reply Last reply Reply Quote 0
                                      • rhessingerR Offline
                                        rhessinger inedo-engineer
                                        last edited by

                                        Hi @jyip_5228,

                                        I just verified and the fix is still in place. We have seen connection issues in the past with the Mono framework due to how connection sharing is implemented in the mono framework. How often does this issue happen? If you restart the ProGet image, how long before you see the connection pool error?

                                        Just to let you know, as part of ProGet 5.3.10 release, we shipped the ProGetCore container image as well.

                                        You can follow the normal steps in the Linux and Docker Installation Guide to install/upgrade, but just use progetcore for the container instead of proget.

                                        Aside from support for the Lucene-based Maven feed indexing (in progress), it seems to be feature complete. And of course, if there are problems, you can switch back to proget:5.3.10 or downgrade as needed (no database schema changes).

                                        For example, docker pull proget.inedo.com/productimages/inedo/progetcore:5.3.10

                                        Thanks,
                                        Rich

                                        Products Engineer, Inedo

                                        1 Reply Last reply Reply Quote 0
                                        • J Offline
                                          jyip_5228
                                          last edited by

                                          I havent tracked it, but i restarted container on Monday, by Tuesday morning when people returned to office/ started running builds, I started seeing Timeout errors. I will switch to the core version and see how that goes.

                                          1 Reply Last reply Reply Quote 0
                                          • rhessingerR Offline
                                            rhessinger inedo-engineer
                                            last edited by

                                            Hi @jyip_5228,

                                            Are you still running the default max connections property? Or have you increased it? Also, how many concurrent builds are you running at a time?

                                            Please send us an update on how the .Net Core version is working out for you.

                                            Thanks,
                                            Rich

                                            Products Engineer, Inedo

                                            1 Reply Last reply Reply Quote 0

                                            Hello! It looks like you're interested in this conversation, but you don't have an account yet.

                                            Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.

                                            With your input, this post could be even better 💗

                                            Register Login
                                            • 1
                                            • 2
                                            • 1 / 2
                                            • First post
                                              Last post
                                            Inedo Website Home • Support Home • Code of Conduct • Forums Guide • Documentation