Welcome to the Inedo Forums! Check out the Forums Guide for help getting started.
If you are experiencing any issues with the forum software, please visit the Contact Form on our website and let us know!
Timeouts connecting to DB when running in Linux
-
Hi,
I have set up a few proget docker containers (web, db), and then started pushing packages to them.
Everything seemed fine, it was running for about 5 hours yesterday.
I come in this morning and the proget web server is down. Logs for the proget web server show:[ERROR] FATAL UNHANDLED EXCEPTION: System.TimeoutException: The operation has timed out. at Npgsql.NpgsqlTimeout.Check () [0x00008] in <4985dea8374f4819bebb7847d95ad483>:0 at Npgsql.NpgsqlConnector.HandleAuthentication (System.String username, Npgsql.NpgsqlTimeout timeout) [0x0001d] in <4985dea8374f4819bebb7847d95ad483>:0 at Npgsql.NpgsqlConnector.Open (Npgsql.NpgsqlTimeout timeout) [0x00071] in <4985dea8374f4819bebb7847d95ad483>:0 at Npgsql.ConnectorPool.Allocate (Npgsql.NpgsqlConnection conn, Npgsql.NpgsqlTimeout timeout) [0x00133] in <4985dea8374f4819bebb7847d95ad483>:0 at Npgsql.NpgsqlConnection.OpenInternal () [0x000e9] in <4985dea8374f4819bebb7847d95ad483>:0 at Npgsql.NpgsqlConnection.Open () [0x00000] in <4985dea8374f4819bebb7847d95ad483>:0 at Inedo.Postgres.PostgresDatabaseContext.CreateConnection () [0x0000b] in <e3135482b02b4587b8331b1f006ab09d>:0 at Inedo.ProGet.Data.ProGetDatabaseContext+WrappedPostgresContext.Inedo.ProGet.Data.ProGetDatabaseContext.IWrappedContext.CreateConnection () [0x00000] in <71613e4cf66848b0a13ff66ec5f0e241>:0 at Inedo.ProGet.Data.ProGetDatabaseContext.CreateConnection () [0x00000] in <71613e4cf66848b0a13ff66ec5f0e241>:0 at Inedo.Data.DatabaseContext.ExecuteInternal (System.String storedProcName, Inedo.Data.GenericDbParameter[] parameters) [0x00058] in <aeac86c668ce402580bc35d6b389d77c>:0 at Inedo.Data.DatabaseContext+<>c__DisplayClass26_0`1[TRow].<EnumerateTable>b__0 () [0x00000] in <aeac86c668ce402580bc35d6b389d77c>:0 at Inedo.Data.StrongDataReader+<Read>d__2`1[TRow].MoveNext () [0x0003e] in <aeac86c668ce402580bc35d6b389d77c>:0 at System.Collections.Generic.List`1[T]..ctor (System.Collections.Generic.IEnumerable`1[T] collection) [0x00077] in <4dc8ec68b0964e099af86e50301f5f3c>:0 at System.Linq.Enumerable.ToList[TSource] (System.Collections.Generic.IEnumerable`1[T] source) [0x00018] in <c716d84e34fc4c408a98d9a0c1024723>:0 at Inedo.ProGet.Data.DB.Configuration_GetConfiguration () [0x0000b] in <71613e4cf66848b0a13ff66ec5f0e241>:0 at Inedo.ProGet.InternalConfig.ReadConfigValues () [0x00000] in <71613e4cf66848b0a13ff66ec5f0e241>:0 at Inedo.LazyCached`1[T].GetValue () [0x00019] in <aeac86c668ce402580bc35d6b389d77c>:0 at Inedo.LazyCached`1[T].get_Value () [0x00000] in <aeac86c668ce402580bc35d6b389d77c>:0 at Inedo.ProGet.ProGetConfig`1[TConfig].ReadFullyQualifiedValue (System.String scopedKey) [0x00000] in <71613e4cf66848b0a13ff66ec5f0e241>:0 at Inedo.ProGet.ProGetConfig`1[TConfig].ReadBoolean (System.String key) [0x00006] in <71613e4cf66848b0a13ff66ec5f0e241>:0 at Inedo.ProGet.ProGetConfig+Web.get_IntegratedAuthenticationEnabled () [0x00000] in <71613e4cf66848b0a13ff66ec5f0e241>:0 at Inedo.ProGet.Web.IntegratedServerAuthenticator.ProGetAuthenticationSchemeSelector (System.Net.HttpListenerRequest httpRequest) [0x0002d] in <782cac368f944d919f2746c16d685751>:0 at System.Net.HttpListener.SelectAuthenticationScheme (System.Net.HttpListenerContext context) [0x00014] in <76fe267e1bf64ad1a7865d07a5edd3f0>:0 at System.Net.ListenerAsyncResult.Complete (System.Net.HttpListenerContext context, System.Boolean synch) [0x00035] in <76fe267e1bf64ad1a7865d07a5edd3f0>:0 at System.Net.ListenerAsyncResult.Complete (System.Net.HttpListenerContext context) [0x00000] in <76fe267e1bf64ad1a7865d07a5edd3f0>:0 at System.Net.HttpListener.RegisterContext (System.Net.HttpListenerContext context) [0x000a6] in <76fe267e1bf64ad1a7865d07a5edd3f0>:0 at System.Net.HttpConnection.OnReadInternal (System.IAsyncResult ares) [0x0014c] in <76fe267e1bf64ad1a7865d07a5edd3f0>:0 at System.Net.HttpConnection.OnRead (System.IAsyncResult ares) [0x00006] in <76fe267e1bf64ad1a7865d07a5edd3f0>:0 at System.Net.Sockets.SocketAsyncResult+<>c__DisplayClass27_0.<Complete>b__0 (System.Object _) [0x00000] in <76fe267e1bf64ad1a7865d07a5edd3f0>:0 at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00008] in <4dc8ec68b0964e099af86e50301f5f3c>:0 at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in <4dc8ec68b0964e099af86e50301f5f3c>:0 at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in <4dc8ec68b0964e099af86e50301f5f3c>:0
Logs for the proget postgres container show:
LOG: could not create IPv6 socket: Address family not supported by protocol LOG: database system was shut down at 2017-06-16 13:07:37 UTC LOG: MultiXact member wraparound protections are now enabled LOG: autovacuum launcher started LOG: database system is ready to accept connections LOG: unexpected EOF on client connection with an open transaction LOG: unexpected EOF on client connection with an open transaction LOG: unexpected EOF on client connection with an open transaction LOG: unexpected EOF on client connection with an open transaction LOG: unexpected EOF on client connection with an open transaction LOG: incomplete startup packet LOG: incomplete startup packet LOG: unexpected EOF on client connection with an open transaction LOG: unexpected EOF on client connection with an open transaction LOG: incomplete startup packet
Restarting either/both of the containers did not resolve the issue. Restarting the docker daemon did help, and I saw that about 50k packages had been uploaded.
4h 15m and another 50k packages later I am getting the same problems.
Product: ProGet
Version: 4.7.13
-
Looks like these symptoms are caused when the host goes OOM