Inedo Community Forums Forums
    • Recent
    • Tags
    • Popular
    • Login

    Welcome to the Inedo Forums! Check out the Forums Guide for help getting started.

    If you are experiencing any issues with the forum software, please visit the Contact Form on our website and let us know!

    Proget Issues (healthcheck, DB connection)

    Scheduled Pinned Locked Moved Support
    15 Posts 3 Posters 51 Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • C Offline
      cshipley_6136
      last edited by

      We had our production instances of proget go down today. One instance was up and running the whole time but the other three all had login issues to the db (another issue as looking at the sql logs there was no failed login) but the /health response said everything was fine. Because the healthcheck was returning okay the servers with issues were never pulled out of the load balancer.

      stevedennisS 1 Reply Last reply Reply Quote 0
      • stevedennisS Offline
        stevedennis inedo-engineer @cshipley_6136
        last edited by

        Hi @cshipley_6136

        Can you give more details about the error messages you were seeing? You mention it's a login error, like a bad SQL password or account or something?

        The health check (/health) runs the Feeds_GetFeeds procedure... so it should have given a 500 error / crashed if the database couldn't be connected.

        We'll try to reproduce the failed failure, and get it fixed asap.

        Cheers,
        Steve

        1 Reply Last reply Reply Quote 0
        • C Offline
          cshipley_6136
          last edited by

          Some details, we are running the instances of proget in containers. Stoping and Starting the container seemed to fix the problem. We did test connectivity to the db server and login creds and it worked fine. While getting the db errors we were getting "databaseStatus": "OK" in the /health check.

          Below is the trace for the db error:

          Logged: 02/08/2023 10:12:17

          Level: Error

          Category: Web

          Message: An error occurred processing a GET request to http://repo.[masked]/nuget/internal-nuget/v3/index.json: Login failed for user '[masked]'.

          Details: Microsoft.Data.SqlClient.SqlException (0x80131904): Login failed for user [masked].
          at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
          at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection) at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
          at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource1 retry, DbConnectionOptions userOptions) at Microsoft.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource1 retry, SqlConnectionOverrides overrides)
          at Microsoft.Data.SqlClient.SqlConnection.Open(SqlConnectionOverrides overrides)
          at Inedo.Data.SqlServerDatabaseContext.CreateConnection()
          at Inedo.Data.DatabaseContext.ExecuteInternal(String storedProcName, GenericDbParameter[] parameters)
          at Inedo.Data.SqlServerDatabaseContext.ExecuteInternal(String storedProcName, GenericDbParameter[] parameters)
          at Inedo.Data.StrongDataReader.Read[TRow](Func1 getReader, Boolean disposeReader)+MoveNext() at Inedo.ProGet.FeedCache.FeedLookups..ctor(IEnumerable1 allFeeds) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp_E281198\Src\ProGetCoreEx\FeedCache.cs:line 90
          at Inedo.ProGet.FeedCache.CreateFeedLookups() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp_E281198\Src\ProGetCoreEx\FeedCache.cs:line 57
          at Inedo.LazyCached1.GetValue() at Inedo.ProGet.FeedCache.GetFeeds(Boolean includeInactive) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E281198\Src\ProGetCoreEx\FeedCache.cs:line 25 at Inedo.ProGet.Web.Security.TaskChecker..ctor() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E281198\Src\ProGetCoreEx\Web\Security\TaskChecker.cs:line 33 at Inedo.ProGet.ProGetSdkConfig.Config.GetCurrentTaskChecker() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E281198\Src\ProGetCoreEx\ProGetSdkConfig.cs:line 81 at Inedo.Security.UserContext.<>c.<InvalidateCache>b__16_0() at System.Lazy1.ViaFactory(LazyThreadSafetyMode mode)
          --- End of stack trace from previous location ---
          at System.Lazy1.CreateValue() at Inedo.ProGet.WebApplication.Security.WebApiContext.IsAuthorizedForTaskInternal(ProGetSecuredTask task, Nullable1 feedId) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp_E281198\Src\ProGet.WebApplication\Security\WebApiContext.cs:line 244
          at Inedo.ProGet.WebApplication.Security.WebApiContext.ValidatePrivileges(ProGetSecuredTask task, Nullable`1 feedId) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp_E281198\Src\ProGet.WebApplication\Security\WebApiContext.cs:line 102
          at Inedo.ProGet.WebApplication.FeedEndpoints.NuGet.V3.ServiceIndexHandler.ProcessRequestAsync(NuGetFeed feed, WebApiContext apiContext, HttpContext context, String relativeUrl) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp_E281198\Src\ProGet.WebApplication\FeedEndpoints\NuGet\V3\ServiceIndexHandler.cs:line 20
          at Inedo.ProGet.WebApplication.FeedEndpoints.NuGet.NuGetFeedHandler.ProcessRequestAsync(HttpContext context, WebApiContext apiContext, NuGetFeed feed, String relativeUrl) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp_E281198\Src\ProGet.WebApplication\FeedEndpoints\NuGet\NuGetFeedHandler.cs:line 43
          at Inedo.ProGet.WebApplication.FeedEndpoints.FeedEndpointHandler.FeedRequestHandler.ProcessRequestAsync(HttpContext context) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp_E281198\Src\ProGet.WebApplication\FeedEndpoints\FeedEndpointHandler.cs:line 197
          ClientConnectionId:aa6efcaf-0204-47cf-bb41-29719fddbacc
          Error Number:18456,State:1,Class:14

          1 Reply Last reply Reply Quote 0
          • C Offline
            cshipley_6136
            last edited by

            Also FWIW, when the instances are down, if we hit them through the load balancer we get some activity as it adds a /error?cachedErrorKey=[masked_just_in_case] to the url.

            1 Reply Last reply Reply Quote 0
            • C Offline
              cshipley_6136
              last edited by

              We are on 2022.14

              atrippA 1 Reply Last reply Reply Quote 0
              • atrippA Offline
                atripp inedo-engineer @cshipley_6136
                last edited by

                Hi @cshipley_6136 ,

                Thanks for the additional information; we were able to figure out what the underlying issue was with your help. Essentially, it a combination of command caching and some other factors that caused this false-positive behavior. The error should have eventually been triggered after a little while, but it's hard to say.

                In any case, we've changed this to use a one minute cache, so this kind of error will be detected much quicker. Either databaseStatus will say Error, or the handler will return a 500 status.

                This will be fixed in PG-2284 , which is scheduled for this Friday's maintenance release (2022.20).

                Cheers,
                Alana

                1 Reply Last reply Reply Quote 0
                • C Offline
                  cshipley_6136
                  last edited by

                  @atripp We have yet to determine why it gets a user auth issue with the database from the trace above. Restarting the container makes everything work. Its almost like something happens and it fails and just keeps reporting a failure and crashing instead of actually retrying? Does it store a session or something? I would think if the database went down for an hour and came back up the application would recover.

                  We are still digging to see if timing works out with database maintenance. In production we use a cluster but there is a blip when the listener flips.

                  atrippA 1 Reply Last reply Reply Quote 0
                  • atrippA Offline
                    atripp inedo-engineer @cshipley_6136
                    last edited by

                    Hi @cshipley_6136 ,

                    In general, when the database has temporary problems (timeouts, unavailability, etc.), then errors will be shown to users (like the above) when they make a web request. Every web request is a new try, basically, and will cause the same error. But once the database is back up, new requests should work fine.

                    However, if the application fails during initialization (i.e. the first web request after starting the service), then it effectively requires restarting the application (container/service). This scenario is tricky to work-around.

                    Otherwise, what was the nature of the user auth issue? If it was an incorrect password, then that would require fixing the connection string (which is passed as an environment variable usually) and then restarting the container.

                    Hope that helps...

                    Alana

                    1 Reply Last reply Reply Quote 0
                    • C Offline
                      cshipley_6136
                      last edited by stevedennis

                      So our development instance is also having the same/similar issue. We have followed to log to show what happens when you try to access it. You can see a bunch of connection errors to sql. We have checked and there are no logged failed attempts to the sql server. Something odd is going on. Here are the logs when it happened from the container:

                      2023-02-10T11:52:44.439594705-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:44.439594705-05:00       Request starting HTTP/1.1 GET http://[masked]/health - 0
2023-02-10T11:52:44.439708824-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:44.439708824-05:00       Request finished HTTP/1.1 GET http://[masked]/health - 0 - 200 492 application/json 0.2093ms
2023-02-10T11:52:44.450096141-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:44.450096141-05:00       Request starting HTTP/1.1 GET http://[masked]/health - 0
2023-02-10T11:52:44.450159808-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:44.450159808-05:00       Request finished HTTP/1.1 GET http://[masked]/health - 0 - 200 492 application/json 0.2018ms
2023-02-10T11:52:44.459275654-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:44.459275654-05:00       Request starting HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=3f518b401a2e47f2bdb867e11b179fd4 - -
2023-02-10T11:52:44.460040115-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:44.460040115-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is c2023-02-10T11:52:44.460064687-05:00 onfigured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:44.461628356-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:44.461628356-05:00       Request finished HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=3f518b401a2e47f2bdb867e11b179fd4 - - - 302 0 text/html;+charset=UTF-8 2.2644ms
2023-02-10T11:52:44.526716795-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:44.526716795-05:00       Request starting HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=4e361390c963474abf6538c45837c453 - -
2023-02-10T11:52:44.527608663-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:44.527608663-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:44.529241678-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:44.529241678-05:00       Request finished HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=4e361390c963474abf6538c45837c453 - - - 302 0 text/html;+charset=UTF-8 2.5601ms
2023-02-10T11:52:44.717331999-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:44.717331999-05:00       Request starting HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=ef6db44421d6454089f45fdf6eec5ac6 - -
2023-02-10T11:52:44.718192986-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:44.718219660-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:44.719886424-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:44.719886424-05:00       Request finished HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=ef6db44421d6454089f45fdf6eec5ac6 - - - 302 0 text/html;+charset=UTF-8 2.3505ms
2023-02-10T11:52:44.879356103-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:44.879356103-05:00       Request starting HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=37d29cc9603c4ed295c17049d537f04b - -
2023-02-10T11:52:44.880298791-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:44.880298791-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:44.882168636-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:44.882168636-05:00       Request finished HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=37d29cc9603c4ed295c17049d537f04b - - - 302 0 text/html;+charset=UTF-8 2.6004ms
2023-02-10T11:52:44.939233556-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:44.939233556-05:00       Request starting HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=7973c67f1448412cbd2d0e3afd9ddfa4 - -
2023-02-10T11:52:44.940100287-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:44.940100287-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:44.941844678-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:44.941844678-05:00       Request finished HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=7973c67f1448412cbd2d0e3afd9ddfa4 - - - 302 0 text/html;+charset=UTF-8 2.4573ms
2023-02-10T11:52:45.039116838-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:45.039116838-05:00       Request starting HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=c3543ed21b7249a0b9c954d0966f9576 - -
2023-02-10T11:52:45.039980090-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.039980090-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is c2023-02-10T11:52:45.040007288-05:00 onfigured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.041646090-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:45.041646090-05:00       Request finished HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=c3543ed21b7249a0b9c954d0966f9576 - - - 302 0 text/html;+charset=UTF-8 2.3688ms
2023-02-10T11:52:45.117314820-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:45.117314820-05:00       Request starting HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=9aca1f59a016400a91631de1618ffa23 - -
2023-02-10T11:52:45.118112060-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.118112060-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is c2023-02-10T11:52:45.118136833-05:00 onfigured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.119702147-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:45.119702147-05:00       Request finished HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=9aca1f59a016400a91631de1618ffa23 - - - 302 0 text/html;+charset=UTF-8 2.3064ms
2023-02-10T11:52:45.189208121-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:45.189208121-05:00       Request starting HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=c563290a97e54301b48f233487397797 - -
2023-02-10T11:52:45.189981802-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.189981802-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is c2023-02-10T11:52:45.190010665-05:00 onfigured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.191657706-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:45.191657706-05:00       Request finished HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=c563290a97e54301b48f233487397797 - - - 302 0 text/html;+charset=UTF-8 2.3746ms
2023-02-10T11:52:45.269237394-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:45.269237394-05:00       Request starting HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=ec0e6b687f4e42d6a3aefc2cb3fb226d - -
2023-02-10T11:52:45.270076468-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.270076468-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is c2023-02-10T11:52:45.270100826-05:00 onfigured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.271810549-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:45.271810549-05:00       Request finished HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=ec0e6b687f4e42d6a3aefc2cb3fb226d - - - 302 0 text/html;+charset=UTF-8 2.4243ms
2023-02-10T11:52:45.339403543-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:45.339403543-05:00       Request starting HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=e1d5b38a2eb44ac788f9a9fe73c197b3 - -
2023-02-10T11:52:45.340209589-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.340209589-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.341973498-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:45.341973498-05:00       Request finished HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=e1d5b38a2eb44ac788f9a9fe73c197b3 - - - 302 0 text/html;+charset=UTF-8 2.4828ms
2023-02-10T11:52:45.429473245-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:45.429473245-05:00       Request starting HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=bb082d582af34298b35e3a594a44cf48 - -
2023-02-10T11:52:45.430269033-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.430293609-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.431919233-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:45.431919233-05:00       Request finished HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=bb082d582af34298b35e3a594a44cf48 - - - 302 0 text/html;+charset=UTF-8 2.3369ms
2023-02-10T11:52:45.509214993-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:45.509214993-05:00       Request starting HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=b742872d950a46f9a4fdf510b21d4964 - -
2023-02-10T11:52:45.510160755-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.510160755-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.511791274-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:45.511791274-05:00       Request finished HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=b742872d950a46f9a4fdf510b21d4964 - - - 302 0 text/html;+charset=UTF-8 2.5908ms
2023-02-10T11:52:45.589480139-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:45.589480139-05:00       Request starting HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=4e792ff4974247ee85b009963b86ba30 - -
2023-02-10T11:52:45.590103868-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.590137304-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.591676546-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:45.591676546-05:00       Request finished HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=4e792ff4974247ee85b009963b86ba30 - - - 302 0 text/html;+charset=UTF-8 2.3383ms
2023-02-10T11:52:45.676875657-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:45.676875657-05:00       Request starting HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=e164331b2b4643968bafea118ecb3adb - -
2023-02-10T11:52:45.677701419-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.677701419-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is c2023-02-10T11:52:45.677723649-05:00 onfigured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.679409112-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:45.679409112-05:00       Request finished HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=e164331b2b4643968bafea118ecb3adb - - - 302 0 text/html;+charset=UTF-8 2.3912ms
2023-02-10T11:52:45.757541776-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:45.757541776-05:00       Request starting HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=664777958ed94583b16a84cd4edeb77b - -
2023-02-10T11:52:45.758397341-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.758397341-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.760001691-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:45.760001691-05:00       Request finished HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=664777958ed94583b16a84cd4edeb77b - - - 302 0 text/html;+charset=UTF-8 2.2949ms
2023-02-10T11:52:45.829441889-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:45.829441889-05:00       Request starting HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=0e305c4572194d06be7be6d58f369c41 - -
2023-02-10T11:52:45.829943598-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.829965899-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.831605830-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:45.831605830-05:00       Request finished HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=0e305c4572194d06be7be6d58f369c41 - - - 302 0 text/html;+charset=UTF-8 2.3670ms
2023-02-10T11:52:46.442792236-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:46.442792236-05:00       Request starting HTTP/1.1 GET http://[masked]/health - 0
2023-02-10T11:52:46.442792236-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:46.442792236-05:00       Request finished HTTP/1.1 GET http://[masked]/health - 0 - 200 492 application/json 0.2448ms
2023-02-10T11:52:46.451216474-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:46.451216474-05:00       Request starting HTTP/1.1 GET http://[masked]/health - 0
2023-02-10T11:52:46.451216474-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:46.451216474-05:00       Request finished HTTP/1.1 GET http://[masked]/health - 0 - 200 492 application/json 0.2091ms
                      stevedennisS 1 Reply Last reply Reply Quote 0
                      • stevedennisS Offline
                        stevedennis inedo-engineer @cshipley_6136
                        last edited by

                        Hi @cshipley_6136 ,

                        Based on this, it looks like there are some network problems between ProGet and your SQL Server:

                        A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections.

                        It's really easy to reproduce this error (just make a typo in the server name) but when it's intermittently happening, it means the problem is happening within the network, between the two servers.

                        Unfortunately you won't be able to find any useful information on the SQL Server side, since the connection cannot be established. And the same is true on the ProGet side, for the same reason.

                        This is definitely something to bring up with your Network/Operations team. It could be anything from DNS issues to router configuration to a bad/failing switch.

                        Best,
                        Steve

                        1 Reply Last reply Reply Quote 0
                        • C Offline
                          cshipley_6136
                          last edited by

                          Hey @stevedennis ,

                          We thought it might be a network issue as well, however it doesn't seem to be. I went into the proget container and did nslookups on the hostname, traceroutes on the hostname and even was able to telnet successfully to the sql server all from within the proget container proget is currently running on. All while proget produces the errors I showed in my previous post. This seems to be some sort of issue somewhere in the application.

                          Connection to the sql server from the container host as well as from inside the proget container.

                          Thoughts?

                          atrippA 1 Reply Last reply Reply Quote 0
                          • atrippA Offline
                            atripp inedo-engineer @cshipley_6136
                            last edited by

                            Hi @cshipley_6136,

                            We aren't trying to pass the buck here, but given the symptoms, it's almost certainly not a software problem I'm afraid. Under the hood, ProGet uses Microsoft's SQL Server driver/components, which uses the operating system's networking drivers/components to communicate to the server. In this case, the error is originating in the operating system's networking components.

                            Based on the symptoms (intermittent network-level errors), it's most certainly a problem with the network hardware, components, or configuration. Since the problem is intermittent, running nslookup similar commands on the container won't really identify any issues; those commands also communicate on different protocols/ports than SQL Server.

                            We're not really experts at troubleshooting network problems, but we have seen a few over the years. Have you had a chance to bring this up with your Network/Operations team? What have they tried or investigated so far?

                            Cheers,
                            Alana

                            1 Reply Last reply Reply Quote 0
                            • C Offline
                              cshipley_6136
                              last edited by cshipley_6136

                              Yes networking/operations has been involved we did external tests and tests from inside the container while proget was running as well. We can telnet to the sql server on the port sql server listens on for connections from inside the container, which shows the sql host on 1433 is indeed able to be found. That is what isn't making sense. Proget errors says it can't reach the server, but from inside the container we can reach it just fine. So only the instance of Proget seems to have the problem. Also doing nothing to the container host, and doing nothing to anything else, simply restarting the container when it does this fixes the problem.

                              1 Reply Last reply Reply Quote 0
                              • C Offline
                                cshipley_6136
                                last edited by cshipley_6136

                                If it makes a difference, we are running this container on OEL8 rootless with podman. So the slirps4netns is installed. So inside the container we see the 10.0.2.3 proxy nameserver, as well as our internal ones being passed down from the host into the container. However, all of that seems to be working and resolving the sql hostname correctly. Checking this config out all looks fine too.

                                stevedennisS 1 Reply Last reply Reply Quote 0
                                • stevedennisS Offline
                                  stevedennis inedo-engineer @cshipley_6136
                                  last edited by

                                  hi @cshipley_6136,

                                  We chatted a bit about this as a team today.

                                  We believe that the application failed during initialization (i.e. the first web request after starting the service); when this happens, the error message will be returned until you restart the application (container/service).

                                  This is by design; there's no way to guess where or why the initialization failed, and if we "crash" the application (and trigger a restart), it can end up in a endless loop and make diagnosing really difficult.

                                  In this case, the error is network related (server name not found). That error can be transient when you have network problems, or permanent if it's a typo in the connection string.

                                  Since restarting the application (container) fixed it, then it was a transient network issue that happened during initialization.

                                  Hope this helps,
                                  Steve

                                  1 Reply Last reply Reply Quote 0

                                  Hello! It looks like you're interested in this conversation, but you don't have an account yet.

                                  Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.

                                  With your input, this post could be even better 💗

                                  Register Login
                                  • 1 / 1
                                  • First post
                                    Last post
                                  Inedo Website Home • Support Home • Code of Conduct • Forums Guide • Documentation