Welcome to the Inedo Forums! Check out the Forums Guide for help getting started.

If you are experiencing any issues with the forum software, please visit the Contact Form on our website and let us know!

Timeouts connecting to DB when running in Linux



  • Hi,

    I have set up a few proget docker containers (web, db), and then started pushing packages to them.

    Everything seemed fine, it was running for about 5 hours yesterday.
    I come in this morning and the proget web server is down. Logs for the proget web server show:

    [ERROR] FATAL UNHANDLED EXCEPTION: System.TimeoutException: The operation has timed out.
      at Npgsql.NpgsqlTimeout.Check () [0x00008] in <4985dea8374f4819bebb7847d95ad483>:0
      at Npgsql.NpgsqlConnector.HandleAuthentication (System.String username, Npgsql.NpgsqlTimeout timeout) [0x0001d] in <4985dea8374f4819bebb7847d95ad483>:0
      at Npgsql.NpgsqlConnector.Open (Npgsql.NpgsqlTimeout timeout) [0x00071] in <4985dea8374f4819bebb7847d95ad483>:0
      at Npgsql.ConnectorPool.Allocate (Npgsql.NpgsqlConnection conn, Npgsql.NpgsqlTimeout timeout) [0x00133] in <4985dea8374f4819bebb7847d95ad483>:0
      at Npgsql.NpgsqlConnection.OpenInternal () [0x000e9] in <4985dea8374f4819bebb7847d95ad483>:0
      at Npgsql.NpgsqlConnection.Open () [0x00000] in <4985dea8374f4819bebb7847d95ad483>:0
      at Inedo.Postgres.PostgresDatabaseContext.CreateConnection () [0x0000b] in <e3135482b02b4587b8331b1f006ab09d>:0
      at Inedo.ProGet.Data.ProGetDatabaseContext+WrappedPostgresContext.Inedo.ProGet.Data.ProGetDatabaseContext.IWrappedContext.CreateConnection () [0x00000] in <71613e4cf66848b0a13ff66ec5f0e241>:0
      at Inedo.ProGet.Data.ProGetDatabaseContext.CreateConnection () [0x00000] in <71613e4cf66848b0a13ff66ec5f0e241>:0
      at Inedo.Data.DatabaseContext.ExecuteInternal (System.String storedProcName, Inedo.Data.GenericDbParameter[] parameters) [0x00058] in <aeac86c668ce402580bc35d6b389d77c>:0
      at Inedo.Data.DatabaseContext+<>c__DisplayClass26_0`1[TRow].<EnumerateTable>b__0 () [0x00000] in <aeac86c668ce402580bc35d6b389d77c>:0
      at Inedo.Data.StrongDataReader+<Read>d__2`1[TRow].MoveNext () [0x0003e] in <aeac86c668ce402580bc35d6b389d77c>:0
      at System.Collections.Generic.List`1[T]..ctor (System.Collections.Generic.IEnumerable`1[T] collection) [0x00077] in <4dc8ec68b0964e099af86e50301f5f3c>:0
      at System.Linq.Enumerable.ToList[TSource] (System.Collections.Generic.IEnumerable`1[T] source) [0x00018] in <c716d84e34fc4c408a98d9a0c1024723>:0
      at Inedo.ProGet.Data.DB.Configuration_GetConfiguration () [0x0000b] in <71613e4cf66848b0a13ff66ec5f0e241>:0
      at Inedo.ProGet.InternalConfig.ReadConfigValues () [0x00000] in <71613e4cf66848b0a13ff66ec5f0e241>:0
      at Inedo.LazyCached`1[T].GetValue () [0x00019] in <aeac86c668ce402580bc35d6b389d77c>:0
      at Inedo.LazyCached`1[T].get_Value () [0x00000] in <aeac86c668ce402580bc35d6b389d77c>:0
      at Inedo.ProGet.ProGetConfig`1[TConfig].ReadFullyQualifiedValue (System.String scopedKey) [0x00000] in <71613e4cf66848b0a13ff66ec5f0e241>:0
      at Inedo.ProGet.ProGetConfig`1[TConfig].ReadBoolean (System.String key) [0x00006] in <71613e4cf66848b0a13ff66ec5f0e241>:0
      at Inedo.ProGet.ProGetConfig+Web.get_IntegratedAuthenticationEnabled () [0x00000] in <71613e4cf66848b0a13ff66ec5f0e241>:0
      at Inedo.ProGet.Web.IntegratedServerAuthenticator.ProGetAuthenticationSchemeSelector (System.Net.HttpListenerRequest httpRequest) [0x0002d] in <782cac368f944d919f2746c16d685751>:0
      at System.Net.HttpListener.SelectAuthenticationScheme (System.Net.HttpListenerContext context) [0x00014] in <76fe267e1bf64ad1a7865d07a5edd3f0>:0
      at System.Net.ListenerAsyncResult.Complete (System.Net.HttpListenerContext context, System.Boolean synch) [0x00035] in <76fe267e1bf64ad1a7865d07a5edd3f0>:0
      at System.Net.ListenerAsyncResult.Complete (System.Net.HttpListenerContext context) [0x00000] in <76fe267e1bf64ad1a7865d07a5edd3f0>:0
      at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext context) [0x000a6] in <76fe267e1bf64ad1a7865d07a5edd3f0>:0
      at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult ares) [0x0014c] in <76fe267e1bf64ad1a7865d07a5edd3f0>:0
      at System.Net.HttpConnection.OnRead (System.IAsyncResult ares) [0x00006] in <76fe267e1bf64ad1a7865d07a5edd3f0>:0
      at System.Net.Sockets.SocketAsyncResult+<>c__DisplayClass27_0.<Complete>b__0 (System.Object _) [0x00000] in <76fe267e1bf64ad1a7865d07a5edd3f0>:0
      at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <4dc8ec68b0964e099af86e50301f5f3c>:0
      at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <4dc8ec68b0964e099af86e50301f5f3c>:0
      at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <4dc8ec68b0964e099af86e50301f5f3c>:0
    

    Logs for the proget postgres container show:

    LOG:  could not create IPv6 socket: Address family not supported by protocol
    LOG:  database system was shut down at 2017-06-16 13:07:37 UTC
    LOG:  MultiXact member wraparound protections are now enabled
    LOG:  autovacuum launcher started
    LOG:  database system is ready to accept connections
    LOG:  unexpected EOF on client connection with an open transaction
    LOG:  unexpected EOF on client connection with an open transaction
    LOG:  unexpected EOF on client connection with an open transaction
    LOG:  unexpected EOF on client connection with an open transaction
    LOG:  unexpected EOF on client connection with an open transaction
    LOG:  incomplete startup packet
    LOG:  incomplete startup packet
    LOG:  unexpected EOF on client connection with an open transaction
    LOG:  unexpected EOF on client connection with an open transaction
    LOG:  incomplete startup packet
    

    Restarting either/both of the containers did not resolve the issue. Restarting the docker daemon did help, and I saw that about 50k packages had been uploaded.

    4h 15m and another 50k packages later I am getting the same problems.

    Product: ProGet
    Version: 4.7.13



  • Looks like these symptoms are caused when the host goes OOM


Log in to reply
 

Inedo Website HomeSupport HomeCode of ConductForums GuideDocumentation