1. Home
  2. Categories
  3. Support
  4. Proget Issues (healthcheck, DB connection)

Welcome to the Inedo Forums! Check out the Forums Guide for help getting started.

If you are experiencing any issues with the forum software, please visit the Contact Form on our website and let us know!

Proget Issues (healthcheck, DB connection)

  • C

    We had our production instances of proget go down today. One instance was up and running the whole time but the other three all had login issues to the db (another issue as looking at the sql logs there was no failed login) but the /health response said everything was fine. Because the healthcheck was returning okay the servers with issues were never pulled out of the load balancer.

    0 stevedennis 1 Reply
  • stevedennis
    inedo-engineer

    Hi @cshipley_6136

    Can you give more details about the error messages you were seeing? You mention it's a login error, like a bad SQL password or account or something?

    The health check (/health) runs the Feeds_GetFeeds procedure... so it should have given a 500 error / crashed if the database couldn't be connected.

    We'll try to reproduce the failed failure, and get it fixed asap.

    Cheers,
    Steve

    0
  • C

    Some details, we are running the instances of proget in containers. Stoping and Starting the container seemed to fix the problem. We did test connectivity to the db server and login creds and it worked fine. While getting the db errors we were getting "databaseStatus": "OK" in the /health check.

    Below is the trace for the db error:

    Logged: 02/08/2023 10:12:17

    Level: Error

    Category: Web

    Message: An error occurred processing a GET request to http://repo.[masked]/nuget/internal-nuget/v3/index.json: Login failed for user '[masked]'.

    Details: Microsoft.Data.SqlClient.SqlException (0x80131904): Login failed for user [masked].
    at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
    at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection) at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
    at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource1 retry, DbConnectionOptions userOptions) at Microsoft.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource1 retry, SqlConnectionOverrides overrides)
    at Microsoft.Data.SqlClient.SqlConnection.Open(SqlConnectionOverrides overrides)
    at Inedo.Data.SqlServerDatabaseContext.CreateConnection()
    at Inedo.Data.DatabaseContext.ExecuteInternal(String storedProcName, GenericDbParameter[] parameters)
    at Inedo.Data.SqlServerDatabaseContext.ExecuteInternal(String storedProcName, GenericDbParameter[] parameters)
    at Inedo.Data.StrongDataReader.Read[TRow](Func1 getReader, Boolean disposeReader)+MoveNext() at Inedo.ProGet.FeedCache.FeedLookups..ctor(IEnumerable1 allFeeds) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp_E281198\Src\ProGetCoreEx\FeedCache.cs:line 90
    at Inedo.ProGet.FeedCache.CreateFeedLookups() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp_E281198\Src\ProGetCoreEx\FeedCache.cs:line 57
    at Inedo.LazyCached1.GetValue() at Inedo.ProGet.FeedCache.GetFeeds(Boolean includeInactive) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E281198\Src\ProGetCoreEx\FeedCache.cs:line 25 at Inedo.ProGet.Web.Security.TaskChecker..ctor() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E281198\Src\ProGetCoreEx\Web\Security\TaskChecker.cs:line 33 at Inedo.ProGet.ProGetSdkConfig.Config.GetCurrentTaskChecker() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E281198\Src\ProGetCoreEx\ProGetSdkConfig.cs:line 81 at Inedo.Security.UserContext.<>c.<InvalidateCache>b__16_0() at System.Lazy1.ViaFactory(LazyThreadSafetyMode mode)
    --- End of stack trace from previous location ---
    at System.Lazy1.CreateValue() at Inedo.ProGet.WebApplication.Security.WebApiContext.IsAuthorizedForTaskInternal(ProGetSecuredTask task, Nullable1 feedId) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp_E281198\Src\ProGet.WebApplication\Security\WebApiContext.cs:line 244
    at Inedo.ProGet.WebApplication.Security.WebApiContext.ValidatePrivileges(ProGetSecuredTask task, Nullable`1 feedId) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp_E281198\Src\ProGet.WebApplication\Security\WebApiContext.cs:line 102
    at Inedo.ProGet.WebApplication.FeedEndpoints.NuGet.V3.ServiceIndexHandler.ProcessRequestAsync(NuGetFeed feed, WebApiContext apiContext, HttpContext context, String relativeUrl) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp_E281198\Src\ProGet.WebApplication\FeedEndpoints\NuGet\V3\ServiceIndexHandler.cs:line 20
    at Inedo.ProGet.WebApplication.FeedEndpoints.NuGet.NuGetFeedHandler.ProcessRequestAsync(HttpContext context, WebApiContext apiContext, NuGetFeed feed, String relativeUrl) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp_E281198\Src\ProGet.WebApplication\FeedEndpoints\NuGet\NuGetFeedHandler.cs:line 43
    at Inedo.ProGet.WebApplication.FeedEndpoints.FeedEndpointHandler.FeedRequestHandler.ProcessRequestAsync(HttpContext context) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp_E281198\Src\ProGet.WebApplication\FeedEndpoints\FeedEndpointHandler.cs:line 197
    ClientConnectionId:aa6efcaf-0204-47cf-bb41-29719fddbacc
    Error Number:18456,State:1,Class:14

    0
  • C

    Also FWIW, when the instances are down, if we hit them through the load balancer we get some activity as it adds a /error?cachedErrorKey=[masked_just_in_case] to the url.

    0
  • C

    We are on 2022.14

    0 atripp 1 Reply
  • atripp
    inedo-engineer

    Hi @cshipley_6136 ,

    Thanks for the additional information; we were able to figure out what the underlying issue was with your help. Essentially, it a combination of command caching and some other factors that caused this false-positive behavior. The error should have eventually been triggered after a little while, but it's hard to say.

    In any case, we've changed this to use a one minute cache, so this kind of error will be detected much quicker. Either databaseStatus will say Error, or the handler will return a 500 status.

    This will be fixed in PG-2284 , which is scheduled for this Friday's maintenance release (2022.20).

    Cheers,
    Alana

    0
  • C

    @atripp We have yet to determine why it gets a user auth issue with the database from the trace above. Restarting the container makes everything work. Its almost like something happens and it fails and just keeps reporting a failure and crashing instead of actually retrying? Does it store a session or something? I would think if the database went down for an hour and came back up the application would recover.

    We are still digging to see if timing works out with database maintenance. In production we use a cluster but there is a blip when the listener flips.

    0 atripp 1 Reply
  • atripp
    inedo-engineer

    Hi @cshipley_6136 ,

    In general, when the database has temporary problems (timeouts, unavailability, etc.), then errors will be shown to users (like the above) when they make a web request. Every web request is a new try, basically, and will cause the same error. But once the database is back up, new requests should work fine.

    However, if the application fails during initialization (i.e. the first web request after starting the service), then it effectively requires restarting the application (container/service). This scenario is tricky to work-around.

    Otherwise, what was the nature of the user auth issue? If it was an incorrect password, then that would require fixing the connection string (which is passed as an environment variable usually) and then restarting the container.

    Hope that helps...

    Alana

    0
  • C

    So our development instance is also having the same/similar issue. We have followed to log to show what happens when you try to access it. You can see a bunch of connection errors to sql. We have checked and there are no logged failed attempts to the sql server. Something odd is going on. Here are the logs when it happened from the container:

    2023-02-10T11:52:44.439594705-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:44.439594705-05:00       Request starting HTTP/1.1 GET http://[masked]/health - 0
2023-02-10T11:52:44.439708824-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:44.439708824-05:00       Request finished HTTP/1.1 GET http://[masked]/health - 0 - 200 492 application/json 0.2093ms
2023-02-10T11:52:44.450096141-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:44.450096141-05:00       Request starting HTTP/1.1 GET http://[masked]/health - 0
2023-02-10T11:52:44.450159808-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:44.450159808-05:00       Request finished HTTP/1.1 GET http://[masked]/health - 0 - 200 492 application/json 0.2018ms
2023-02-10T11:52:44.459275654-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:44.459275654-05:00       Request starting HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=3f518b401a2e47f2bdb867e11b179fd4 - -
2023-02-10T11:52:44.460040115-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:44.460040115-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is c2023-02-10T11:52:44.460064687-05:00 onfigured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:44.461628356-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:44.461628356-05:00       Request finished HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=3f518b401a2e47f2bdb867e11b179fd4 - - - 302 0 text/html;+charset=UTF-8 2.2644ms
2023-02-10T11:52:44.526716795-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:44.526716795-05:00       Request starting HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=4e361390c963474abf6538c45837c453 - -
2023-02-10T11:52:44.527608663-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:44.527608663-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:44.529241678-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:44.529241678-05:00       Request finished HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=4e361390c963474abf6538c45837c453 - - - 302 0 text/html;+charset=UTF-8 2.5601ms
2023-02-10T11:52:44.717331999-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:44.717331999-05:00       Request starting HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=ef6db44421d6454089f45fdf6eec5ac6 - -
2023-02-10T11:52:44.718192986-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:44.718219660-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:44.719886424-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:44.719886424-05:00       Request finished HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=ef6db44421d6454089f45fdf6eec5ac6 - - - 302 0 text/html;+charset=UTF-8 2.3505ms
2023-02-10T11:52:44.879356103-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:44.879356103-05:00       Request starting HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=37d29cc9603c4ed295c17049d537f04b - -
2023-02-10T11:52:44.880298791-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:44.880298791-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:44.882168636-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:44.882168636-05:00       Request finished HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=37d29cc9603c4ed295c17049d537f04b - - - 302 0 text/html;+charset=UTF-8 2.6004ms
2023-02-10T11:52:44.939233556-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:44.939233556-05:00       Request starting HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=7973c67f1448412cbd2d0e3afd9ddfa4 - -
2023-02-10T11:52:44.940100287-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:44.940100287-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:44.941844678-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:44.941844678-05:00       Request finished HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=7973c67f1448412cbd2d0e3afd9ddfa4 - - - 302 0 text/html;+charset=UTF-8 2.4573ms
2023-02-10T11:52:45.039116838-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:45.039116838-05:00       Request starting HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=c3543ed21b7249a0b9c954d0966f9576 - -
2023-02-10T11:52:45.039980090-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.039980090-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is c2023-02-10T11:52:45.040007288-05:00 onfigured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.041646090-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:45.041646090-05:00       Request finished HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=c3543ed21b7249a0b9c954d0966f9576 - - - 302 0 text/html;+charset=UTF-8 2.3688ms
2023-02-10T11:52:45.117314820-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:45.117314820-05:00       Request starting HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=9aca1f59a016400a91631de1618ffa23 - -
2023-02-10T11:52:45.118112060-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.118112060-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is c2023-02-10T11:52:45.118136833-05:00 onfigured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.119702147-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:45.119702147-05:00       Request finished HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=9aca1f59a016400a91631de1618ffa23 - - - 302 0 text/html;+charset=UTF-8 2.3064ms
2023-02-10T11:52:45.189208121-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:45.189208121-05:00       Request starting HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=c563290a97e54301b48f233487397797 - -
2023-02-10T11:52:45.189981802-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.189981802-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is c2023-02-10T11:52:45.190010665-05:00 onfigured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.191657706-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:45.191657706-05:00       Request finished HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=c563290a97e54301b48f233487397797 - - - 302 0 text/html;+charset=UTF-8 2.3746ms
2023-02-10T11:52:45.269237394-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:45.269237394-05:00       Request starting HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=ec0e6b687f4e42d6a3aefc2cb3fb226d - -
2023-02-10T11:52:45.270076468-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.270076468-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is c2023-02-10T11:52:45.270100826-05:00 onfigured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.271810549-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:45.271810549-05:00       Request finished HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=ec0e6b687f4e42d6a3aefc2cb3fb226d - - - 302 0 text/html;+charset=UTF-8 2.4243ms
2023-02-10T11:52:45.339403543-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:45.339403543-05:00       Request starting HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=e1d5b38a2eb44ac788f9a9fe73c197b3 - -
2023-02-10T11:52:45.340209589-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.340209589-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.341973498-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:45.341973498-05:00       Request finished HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=e1d5b38a2eb44ac788f9a9fe73c197b3 - - - 302 0 text/html;+charset=UTF-8 2.4828ms
2023-02-10T11:52:45.429473245-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:45.429473245-05:00       Request starting HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=bb082d582af34298b35e3a594a44cf48 - -
2023-02-10T11:52:45.430269033-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.430293609-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.431919233-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:45.431919233-05:00       Request finished HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=bb082d582af34298b35e3a594a44cf48 - - - 302 0 text/html;+charset=UTF-8 2.3369ms
2023-02-10T11:52:45.509214993-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:45.509214993-05:00       Request starting HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=b742872d950a46f9a4fdf510b21d4964 - -
2023-02-10T11:52:45.510160755-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.510160755-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.511791274-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:45.511791274-05:00       Request finished HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=b742872d950a46f9a4fdf510b21d4964 - - - 302 0 text/html;+charset=UTF-8 2.5908ms
2023-02-10T11:52:45.589480139-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:45.589480139-05:00       Request starting HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=4e792ff4974247ee85b009963b86ba30 - -
2023-02-10T11:52:45.590103868-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.590137304-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.591676546-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:45.591676546-05:00       Request finished HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=4e792ff4974247ee85b009963b86ba30 - - - 302 0 text/html;+charset=UTF-8 2.3383ms
2023-02-10T11:52:45.676875657-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:45.676875657-05:00       Request starting HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=e164331b2b4643968bafea118ecb3adb - -
2023-02-10T11:52:45.677701419-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.677701419-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is c2023-02-10T11:52:45.677723649-05:00 onfigured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.679409112-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:45.679409112-05:00       Request finished HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=e164331b2b4643968bafea118ecb3adb - - - 302 0 text/html;+charset=UTF-8 2.3912ms
2023-02-10T11:52:45.757541776-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:45.757541776-05:00       Request starting HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=664777958ed94583b16a84cd4edeb77b - -
2023-02-10T11:52:45.758397341-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.758397341-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.760001691-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:45.760001691-05:00       Request finished HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=664777958ed94583b16a84cd4edeb77b - - - 302 0 text/html;+charset=UTF-8 2.2949ms
2023-02-10T11:52:45.829441889-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:45.829441889-05:00       Request starting HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=0e305c4572194d06be7be6d58f369c41 - -
2023-02-10T11:52:45.829943598-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.829965899-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.831605830-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:45.831605830-05:00       Request finished HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=0e305c4572194d06be7be6d58f369c41 - - - 302 0 text/html;+charset=UTF-8 2.3670ms
2023-02-10T11:52:46.442792236-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:46.442792236-05:00       Request starting HTTP/1.1 GET http://[masked]/health - 0
2023-02-10T11:52:46.442792236-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:46.442792236-05:00       Request finished HTTP/1.1 GET http://[masked]/health - 0 - 200 492 application/json 0.2448ms
2023-02-10T11:52:46.451216474-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:46.451216474-05:00       Request starting HTTP/1.1 GET http://[masked]/health - 0
2023-02-10T11:52:46.451216474-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:46.451216474-05:00       Request finished HTTP/1.1 GET http://[masked]/health - 0 - 200 492 application/json 0.2091ms
    0 stevedennis 1 Reply
  • stevedennis
    inedo-engineer

    Hi @cshipley_6136 ,

    Based on this, it looks like there are some network problems between ProGet and your SQL Server:

    A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections.

    It's really easy to reproduce this error (just make a typo in the server name) but when it's intermittently happening, it means the problem is happening within the network, between the two servers.

    Unfortunately you won't be able to find any useful information on the SQL Server side, since the connection cannot be established. And the same is true on the ProGet side, for the same reason.

    This is definitely something to bring up with your Network/Operations team. It could be anything from DNS issues to router configuration to a bad/failing switch.

    Best,
    Steve

    0
  • C

    Hey @stevedennis ,

    We thought it might be a network issue as well, however it doesn't seem to be. I went into the proget container and did nslookups on the hostname, traceroutes on the hostname and even was able to telnet successfully to the sql server all from within the proget container proget is currently running on. All while proget produces the errors I showed in my previous post. This seems to be some sort of issue somewhere in the application.

    Connection to the sql server from the container host as well as from inside the proget container.

    Thoughts?

    0 atripp 1 Reply
  • atripp
    inedo-engineer

    Hi @cshipley_6136,

    We aren't trying to pass the buck here, but given the symptoms, it's almost certainly not a software problem I'm afraid. Under the hood, ProGet uses Microsoft's SQL Server driver/components, which uses the operating system's networking drivers/components to communicate to the server. In this case, the error is originating in the operating system's networking components.

    Based on the symptoms (intermittent network-level errors), it's most certainly a problem with the network hardware, components, or configuration. Since the problem is intermittent, running nslookup similar commands on the container won't really identify any issues; those commands also communicate on different protocols/ports than SQL Server.

    We're not really experts at troubleshooting network problems, but we have seen a few over the years. Have you had a chance to bring this up with your Network/Operations team? What have they tried or investigated so far?

    Cheers,
    Alana

    0
  • C

    Yes networking/operations has been involved we did external tests and tests from inside the container while proget was running as well. We can telnet to the sql server on the port sql server listens on for connections from inside the container, which shows the sql host on 1433 is indeed able to be found. That is what isn't making sense. Proget errors says it can't reach the server, but from inside the container we can reach it just fine. So only the instance of Proget seems to have the problem. Also doing nothing to the container host, and doing nothing to anything else, simply restarting the container when it does this fixes the problem.

    0
  • C

    If it makes a difference, we are running this container on OEL8 rootless with podman. So the slirps4netns is installed. So inside the container we see the 10.0.2.3 proxy nameserver, as well as our internal ones being passed down from the host into the container. However, all of that seems to be working and resolving the sql hostname correctly. Checking this config out all looks fine too.

    0 stevedennis 1 Reply
  • stevedennis
    inedo-engineer

    hi @cshipley_6136,

    We chatted a bit about this as a team today.

    We believe that the application failed during initialization (i.e. the first web request after starting the service); when this happens, the error message will be returned until you restart the application (container/service).

    This is by design; there's no way to guess where or why the initialization failed, and if we "crash" the application (and trigger a restart), it can end up in a endless loop and make diagnosing really difficult.

    In this case, the error is network related (server name not found). That error can be transient when you have network problems, or permanent if it's a typo in the connection string.

    Since restarting the application (container) fixed it, then it was a transient network issue that happened during initialization.

    Hope this helps,
    Steve

    0
Log in to reply
 
Inedo Website Home • Support Home • Code of Conduct • Forums Guide • Documentation