Posts made by cshipley_6136

@atripp So what do you guys recommend for HA Proxy to do a health check? What is causing the connection error to SQL seems to be a separate issue than the /health endpoint not reporting accurately as everything in the /health endpoint object is OK. The code you posed seems to do it's own call to the the get_feeds proc, that leads me to think that the connection its getting from the connection pool for the health check is working fine but somewhere in the application it's getting an error. Can something be added to the health check to account for this scenario? Like maybe a flag that gets set on these errors that the health check endpoint also looks at?

Here are some other details of the error that may help. There seem to be two errors happening at once, they have the same timestamp in logs. They are probably related/the same.

Category: WEB

Message: An error occurred processing a GET request to http://subdomain.tld/nuget/approved-nuget/v3/registrations-gz/system.security.cryptography.algorithms/index.json: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)

Details: Microsoft.Data.SqlClient.SqlException (0x80131904): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
at 
Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
at 
Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
at 
Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
at 
Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
at Microsoft.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry, SqlConnectionOverrides overrides)
at 
Microsoft.Data.SqlClient.SqlConnection.InternalOpenAsync(CancellationToken cancellationToken)
--- End of stack trace from previous location ---
at 
Inedo.Data.SqlServerDatabaseContext.CreateConnectionAsync()
at 
Inedo.Data.DatabaseContext.ExecuteInternalAsync(String storedProcName, GenericDbParameter[] parameters)
at 
Inedo.Data.SqlServerDatabaseContext.ExecuteInternalAsync(String storedProcName, GenericDbParameter[] parameters)
at 
Inedo.Data.DatabaseContext.ExecuteTableAsync[TRow](String storedProcName, GenericDbParameter[] parameters)
at 
Inedo.ProGet.Feeds.NuGet.NuGetFeed.GetPackageV3Async(String packageId) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E340542\Src\ProGetCoreEx\Feeds\NuGet\NuGetFeed.cs:line 230
at
Inedo.ProGet.WebApplication.FeedEndpoints.NuGet.V3.PackageMetadataHandler.ProcessRequestAsync(NuGetFeed feed, AhHttpContext context, WebApiContext apiContext, String relativeUrl) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E340542\Src\ProGet.WebApplication\FeedEndpoints\NuGet\V3\PackageMetadataHandler.cs:line 35
at 
Inedo.ProGet.WebApplication.FeedEndpoints.NuGet.NuGetFeedHandler.ProcessRequestAsync(AhHttpContext context, WebApiContext apiContext, NuGetFeed feed, String relativeUrl) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E340542\Src\ProGet.WebApplication\FeedEndpoints\NuGet\NuGetFeedHandler.cs:line 30
at 
Inedo.ProGet.WebApplication.FeedEndpoints.FeedEndpointHandler.FeedRequestHandler.ProcessRequestAsync(AhHttpContext context) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E340542\Src\ProGet.WebApplication\FeedEndpoints\FeedEndpointHandler.cs:line 188
ClientConnectionId:00000000-0000-0000-0000-000000000000

Category: Web

Message: An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)

Details:
URL: http://subdomain.tld/nuget/approved-nuget/v3/registrations-gz/microsoft.identitymodel.logging/index.json
Referrer: (not set)
User: (unknown)
User Agent: NuGet VS VSIX/6.7.0 (Microsoft Windows NT 10.0.19045.0, Professional/17.0)
Stack trace: at 
Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
at 
Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)
at 
Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
at 
Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
at 
Microsoft.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry, SqlConnectionOverrides overrides)
at 
Microsoft.Data.SqlClient.SqlConnection.Open(SqlConnectionOverrides overrides)
at 
Inedo.Data.SqlServerDatabaseContext.CreateConnection()
at 
Inedo.Data.DatabaseContext.ExecuteInternal(String storedProcName, GenericDbParameter[] parameters)
at 
Inedo.Data.SqlServerDatabaseContext.ExecuteInternal(String storedProcName, GenericDbParameter[] parameters)
at 
Inedo.Data.StrongDataReader.Read[TRow](Func`1 getReader, Boolean disposeReader)+MoveNext()
at 
System.Linq.Enumerable.TryGetFirst[TSource](IEnumerable`1 source, Boolean& found)
at 
Inedo.ProGet.Data.DB.Context.Feeds_GetFeed(Nullable`1 Feed_Id, String Feed_Name) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E340542\Src\ProGetCoreEx\InedoLib.Analyzers\InedoLib.Analyzers.DatabaseContextGenerator\DB.g.cs:line 5726
at 
Inedo.ProGet.WebApplication.FeedEndpoints.FeedEndpointHandler.GetHttpHandler(GetHandlerParams args) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E340542\Src\ProGet.WebApplication\FeedEndpoints\FeedEndpointHandler.cs:line 43
at 
Inedo.Web.Handlers.DynamicHttpHandling.GetHandler(AhHttpContext context, String requestType, String url, String pathTranslated)
at Inedo.Web.AhWebMiddleware.InvokeAsync(HttpContext context)

Alright, so we have setup a dedicated listener for the proget database and updated the connection string accordingly. We have also added the MultiSubnetFailover=True parameter to the connection string as well. It has been running like this for a couple weeks.

The interesting thing is Today we had an issue with one of the instances where / was showing a 500 error with the following message "A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server) For more information, visit the Error Log Page."

While the / was showing this, the /health endpoint was showing Database: OK and so did the /health?dbcache=fase

So nothing pulled this instance out of the pool on the load balancer.

What's odd is nothing really change infrastructure wise to cause a connectivity problem.

All that being said, the biggest issue we need to resolve is what is the proper way to do health check to ensure if there is an error it does indeed get pulled out of the backend of the load balancer as the /health endpoint doesn't seem to give accurate information.

I am also seeing the following in logs happening every 30 seconds or so.

2023-11-28T15:23:04.177166423-05:00 Running Failover Detection...
2023-11-28T15:23:04.177166423-05:00 Running Failover Detection...
2023-11-28T15:23:04.179161015-05:00 Failover Detection completed.
2023-11-28T15:23:04.179161015-05:00 Failover Detection completed.

Hello,

We have posted about this issue before but would like to post about it again as we noticed something different. When doing maintenance on the SQL server and flipping the primary in the Availability group, it seems like an instance of proget gets hung up. Some come back fine, some get stuck on a message about the database being in an availability group. When this happens, the main page of proget shows a 500 error with the error message (which is great), however, on that same instance while the main page returns the 500, the /health endpoint returns a 200 and says the database status is 'ok'. It even shows that as being the case with the dbcache=false option.

It almost seems like Proget has one connection to the database and the healthcheck has another so the healthcheck isn't actually reporting anything when proget itself has a problem. If this is going to be consistant behavior we can just hit / for our healthcheck in HAProxy instead of /health but that seems counter-intuitive.

Thanks!

@reincarnator247_4909 If you open a ticket can you post it here so I can follow along as well. Or do we have a solution? To me, a call the the /health endpoint should always actually check if things are okay, not assume its okay because something is cached.

So we are having another connection issue, we figured out this was the same as reported before.

We run four instances of proget in containers for HA. Two of the four instances showed errors during the switchover and sync and recovered fine. The whole cluster seemed healthy after that. Upon further investigation, the other two instances were showing some errors hours after the fact. One was getting a login error and the other was talking about the server being in an non-accessable state due to the availability group. It's like they had open connections and wouldn't switch over when the listener was pointed back to the primary sql server. I do have logs showing such.

This exposes another large issue with the proget health check and cluster health. If I logged into one of the working instances it reported that all nodes were healthy and the cluster had no issues. Also, on the two instances that currently could not connect to the database and were getting sql errors the /health endpoint was still reporting database: OK. This causes the whole service to go down because since the healthcheck is okay, there is nothing that would be able to pull those two instances out of the load balancer pool. If the healthcheck gave a 500 or reported database error or something, this error wouldn't have taken our cluster down.

Here are some log entries of the errors. It's odd because all four instances use the same connection string etc but only two have a problem and both were showing a slightly different error.

I am not able to post logs here due to company privacy of hostnames and urls. If there is a support ticket opened I can send them there.

Thanks in advance for the support!

If it makes a difference, we are running this container on OEL8 rootless with podman. So the slirps4netns is installed. So inside the container we see the 10.0.2.3 proxy nameserver, as well as our internal ones being passed down from the host into the container. However, all of that seems to be working and resolving the sql hostname correctly. Checking this config out all looks fine too.

Yes networking/operations has been involved we did external tests and tests from inside the container while proget was running as well. We can telnet to the sql server on the port sql server listens on for connections from inside the container, which shows the sql host on 1433 is indeed able to be found. That is what isn't making sense. Proget errors says it can't reach the server, but from inside the container we can reach it just fine. So only the instance of Proget seems to have the problem. Also doing nothing to the container host, and doing nothing to anything else, simply restarting the container when it does this fixes the problem.

Hey @stevedennis ,

We thought it might be a network issue as well, however it doesn't seem to be. I went into the proget container and did nslookups on the hostname, traceroutes on the hostname and even was able to telnet successfully to the sql server all from within the proget container proget is currently running on. All while proget produces the errors I showed in my previous post. This seems to be some sort of issue somewhere in the application.

Connection to the sql server from the container host as well as from inside the proget container.

Thoughts?

So our development instance is also having the same/similar issue. We have followed to log to show what happens when you try to access it. You can see a bunch of connection errors to sql. We have checked and there are no logged failed attempts to the sql server. Something odd is going on. Here are the logs when it happened from the container:

2023-02-10T11:52:44.439594705-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:44.439594705-05:00       Request starting HTTP/1.1 GET http://[masked]/health - 0
2023-02-10T11:52:44.439708824-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:44.439708824-05:00       Request finished HTTP/1.1 GET http://[masked]/health - 0 - 200 492 application/json 0.2093ms
2023-02-10T11:52:44.450096141-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:44.450096141-05:00       Request starting HTTP/1.1 GET http://[masked]/health - 0
2023-02-10T11:52:44.450159808-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:44.450159808-05:00       Request finished HTTP/1.1 GET http://[masked]/health - 0 - 200 492 application/json 0.2018ms
2023-02-10T11:52:44.459275654-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:44.459275654-05:00       Request starting HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=3f518b401a2e47f2bdb867e11b179fd4 - -
2023-02-10T11:52:44.460040115-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:44.460040115-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is c2023-02-10T11:52:44.460064687-05:00 onfigured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:44.461628356-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:44.461628356-05:00       Request finished HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=3f518b401a2e47f2bdb867e11b179fd4 - - - 302 0 text/html;+charset=UTF-8 2.2644ms
2023-02-10T11:52:44.526716795-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:44.526716795-05:00       Request starting HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=4e361390c963474abf6538c45837c453 - -
2023-02-10T11:52:44.527608663-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:44.527608663-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:44.529241678-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:44.529241678-05:00       Request finished HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=4e361390c963474abf6538c45837c453 - - - 302 0 text/html;+charset=UTF-8 2.5601ms
2023-02-10T11:52:44.717331999-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:44.717331999-05:00       Request starting HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=ef6db44421d6454089f45fdf6eec5ac6 - -
2023-02-10T11:52:44.718192986-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:44.718219660-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:44.719886424-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:44.719886424-05:00       Request finished HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=ef6db44421d6454089f45fdf6eec5ac6 - - - 302 0 text/html;+charset=UTF-8 2.3505ms
2023-02-10T11:52:44.879356103-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:44.879356103-05:00       Request starting HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=37d29cc9603c4ed295c17049d537f04b - -
2023-02-10T11:52:44.880298791-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:44.880298791-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:44.882168636-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:44.882168636-05:00       Request finished HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=37d29cc9603c4ed295c17049d537f04b - - - 302 0 text/html;+charset=UTF-8 2.6004ms
2023-02-10T11:52:44.939233556-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:44.939233556-05:00       Request starting HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=7973c67f1448412cbd2d0e3afd9ddfa4 - -
2023-02-10T11:52:44.940100287-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:44.940100287-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:44.941844678-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:44.941844678-05:00       Request finished HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=7973c67f1448412cbd2d0e3afd9ddfa4 - - - 302 0 text/html;+charset=UTF-8 2.4573ms
2023-02-10T11:52:45.039116838-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:45.039116838-05:00       Request starting HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=c3543ed21b7249a0b9c954d0966f9576 - -
2023-02-10T11:52:45.039980090-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.039980090-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is c2023-02-10T11:52:45.040007288-05:00 onfigured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.041646090-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:45.041646090-05:00       Request finished HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=c3543ed21b7249a0b9c954d0966f9576 - - - 302 0 text/html;+charset=UTF-8 2.3688ms
2023-02-10T11:52:45.117314820-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:45.117314820-05:00       Request starting HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=9aca1f59a016400a91631de1618ffa23 - -
2023-02-10T11:52:45.118112060-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.118112060-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is c2023-02-10T11:52:45.118136833-05:00 onfigured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.119702147-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:45.119702147-05:00       Request finished HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=9aca1f59a016400a91631de1618ffa23 - - - 302 0 text/html;+charset=UTF-8 2.3064ms
2023-02-10T11:52:45.189208121-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:45.189208121-05:00       Request starting HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=c563290a97e54301b48f233487397797 - -
2023-02-10T11:52:45.189981802-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.189981802-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is c2023-02-10T11:52:45.190010665-05:00 onfigured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.191657706-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:45.191657706-05:00       Request finished HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=c563290a97e54301b48f233487397797 - - - 302 0 text/html;+charset=UTF-8 2.3746ms
2023-02-10T11:52:45.269237394-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:45.269237394-05:00       Request starting HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=ec0e6b687f4e42d6a3aefc2cb3fb226d - -
2023-02-10T11:52:45.270076468-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.270076468-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is c2023-02-10T11:52:45.270100826-05:00 onfigured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.271810549-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:45.271810549-05:00       Request finished HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=ec0e6b687f4e42d6a3aefc2cb3fb226d - - - 302 0 text/html;+charset=UTF-8 2.4243ms
2023-02-10T11:52:45.339403543-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:45.339403543-05:00       Request starting HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=e1d5b38a2eb44ac788f9a9fe73c197b3 - -
2023-02-10T11:52:45.340209589-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.340209589-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.341973498-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:45.341973498-05:00       Request finished HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=e1d5b38a2eb44ac788f9a9fe73c197b3 - - - 302 0 text/html;+charset=UTF-8 2.4828ms
2023-02-10T11:52:45.429473245-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:45.429473245-05:00       Request starting HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=bb082d582af34298b35e3a594a44cf48 - -
2023-02-10T11:52:45.430269033-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.430293609-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.431919233-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:45.431919233-05:00       Request finished HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=bb082d582af34298b35e3a594a44cf48 - - - 302 0 text/html;+charset=UTF-8 2.3369ms
2023-02-10T11:52:45.509214993-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:45.509214993-05:00       Request starting HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=b742872d950a46f9a4fdf510b21d4964 - -
2023-02-10T11:52:45.510160755-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.510160755-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.511791274-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:45.511791274-05:00       Request finished HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=b742872d950a46f9a4fdf510b21d4964 - - - 302 0 text/html;+charset=UTF-8 2.5908ms
2023-02-10T11:52:45.589480139-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:45.589480139-05:00       Request starting HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=4e792ff4974247ee85b009963b86ba30 - -
2023-02-10T11:52:45.590103868-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.590137304-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.591676546-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:45.591676546-05:00       Request finished HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=4e792ff4974247ee85b009963b86ba30 - - - 302 0 text/html;+charset=UTF-8 2.3383ms
2023-02-10T11:52:45.676875657-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:45.676875657-05:00       Request starting HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=e164331b2b4643968bafea118ecb3adb - -
2023-02-10T11:52:45.677701419-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.677701419-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is c2023-02-10T11:52:45.677723649-05:00 onfigured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.679409112-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:45.679409112-05:00       Request finished HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=e164331b2b4643968bafea118ecb3adb - - - 302 0 text/html;+charset=UTF-8 2.3912ms
2023-02-10T11:52:45.757541776-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:45.757541776-05:00       Request starting HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=664777958ed94583b16a84cd4edeb77b - -
2023-02-10T11:52:45.758397341-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.758397341-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.760001691-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:45.760001691-05:00       Request finished HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=664777958ed94583b16a84cd4edeb77b - - - 302 0 text/html;+charset=UTF-8 2.2949ms
2023-02-10T11:52:45.829441889-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:45.829441889-05:00       Request starting HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=0e305c4572194d06be7be6d58f369c41 - -
2023-02-10T11:52:45.829943598-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.829965899-05:00 An error occurred in the web application: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)
2023-02-10T11:52:45.831605830-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:45.831605830-05:00       Request finished HTTP/1.1 GET http://repo.dev.[masked]/error?cachedErrorKey=0e305c4572194d06be7be6d58f369c41 - - - 302 0 text/html;+charset=UTF-8 2.3670ms
2023-02-10T11:52:46.442792236-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:46.442792236-05:00       Request starting HTTP/1.1 GET http://[masked]/health - 0
2023-02-10T11:52:46.442792236-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:46.442792236-05:00       Request finished HTTP/1.1 GET http://[masked]/health - 0 - 200 492 application/json 0.2448ms
2023-02-10T11:52:46.451216474-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
2023-02-10T11:52:46.451216474-05:00       Request starting HTTP/1.1 GET http://[masked]/health - 0
2023-02-10T11:52:46.451216474-05:00 info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
2023-02-10T11:52:46.451216474-05:00       Request finished HTTP/1.1 GET http://[masked]/health - 0 - 200 492 application/json 0.2091ms

@atripp We have yet to determine why it gets a user auth issue with the database from the trace above. Restarting the container makes everything work. Its almost like something happens and it fails and just keeps reporting a failure and crashing instead of actually retrying? Does it store a session or something? I would think if the database went down for an hour and came back up the application would recover.

We are still digging to see if timing works out with database maintenance. In production we use a cluster but there is a blip when the listener flips.

We are on 2022.14

Also FWIW, when the instances are down, if we hit them through the load balancer we get some activity as it adds a /error?cachedErrorKey=[masked_just_in_case] to the url.

Some details, we are running the instances of proget in containers. Stoping and Starting the container seemed to fix the problem. We did test connectivity to the db server and login creds and it worked fine. While getting the db errors we were getting "databaseStatus": "OK" in the /health check.

Below is the trace for the db error:

Logged: 02/08/2023 10:12:17

Level: Error

Category: Web

Message: An error occurred processing a GET request to http://repo.[masked]/nuget/internal-nuget/v3/index.json: Login failed for user '[masked]'.

Details: Microsoft.Data.SqlClient.SqlException (0x80131904): Login failed for user [masked].
at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)
at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection) at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource1 retry, DbConnectionOptions userOptions) at Microsoft.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource1 retry, SqlConnectionOverrides overrides)
at Microsoft.Data.SqlClient.SqlConnection.Open(SqlConnectionOverrides overrides)
at Inedo.Data.SqlServerDatabaseContext.CreateConnection()
at Inedo.Data.DatabaseContext.ExecuteInternal(String storedProcName, GenericDbParameter[] parameters)
at Inedo.Data.SqlServerDatabaseContext.ExecuteInternal(String storedProcName, GenericDbParameter[] parameters)
at Inedo.Data.StrongDataReader.Read[TRow](Func1 getReader, Boolean disposeReader)+MoveNext() at Inedo.ProGet.FeedCache.FeedLookups..ctor(IEnumerable1 allFeeds) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp_E281198\Src\ProGetCoreEx\FeedCache.cs:line 90
at Inedo.ProGet.FeedCache.CreateFeedLookups() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp_E281198\Src\ProGetCoreEx\FeedCache.cs:line 57
at Inedo.LazyCached1.GetValue() at Inedo.ProGet.FeedCache.GetFeeds(Boolean includeInactive) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E281198\Src\ProGetCoreEx\FeedCache.cs:line 25 at Inedo.ProGet.Web.Security.TaskChecker..ctor() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E281198\Src\ProGetCoreEx\Web\Security\TaskChecker.cs:line 33 at Inedo.ProGet.ProGetSdkConfig.Config.GetCurrentTaskChecker() in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E281198\Src\ProGetCoreEx\ProGetSdkConfig.cs:line 81 at Inedo.Security.UserContext.<>c.<InvalidateCache>b__16_0() at System.Lazy1.ViaFactory(LazyThreadSafetyMode mode)
--- End of stack trace from previous location ---
at System.Lazy1.CreateValue() at Inedo.ProGet.WebApplication.Security.WebApiContext.IsAuthorizedForTaskInternal(ProGetSecuredTask task, Nullable1 feedId) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp_E281198\Src\ProGet.WebApplication\Security\WebApiContext.cs:line 244
at Inedo.ProGet.WebApplication.Security.WebApiContext.ValidatePrivileges(ProGetSecuredTask task, Nullable`1 feedId) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp_E281198\Src\ProGet.WebApplication\Security\WebApiContext.cs:line 102
at Inedo.ProGet.WebApplication.FeedEndpoints.NuGet.V3.ServiceIndexHandler.ProcessRequestAsync(NuGetFeed feed, WebApiContext apiContext, HttpContext context, String relativeUrl) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp_E281198\Src\ProGet.WebApplication\FeedEndpoints\NuGet\V3\ServiceIndexHandler.cs:line 20
at Inedo.ProGet.WebApplication.FeedEndpoints.NuGet.NuGetFeedHandler.ProcessRequestAsync(HttpContext context, WebApiContext apiContext, NuGetFeed feed, String relativeUrl) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp_E281198\Src\ProGet.WebApplication\FeedEndpoints\NuGet\NuGetFeedHandler.cs:line 43
at Inedo.ProGet.WebApplication.FeedEndpoints.FeedEndpointHandler.FeedRequestHandler.ProcessRequestAsync(HttpContext context) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp_E281198\Src\ProGet.WebApplication\FeedEndpoints\FeedEndpointHandler.cs:line 197
ClientConnectionId:aa6efcaf-0204-47cf-bb41-29719fddbacc
Error Number:18456,State:1,Class:14

We had our production instances of proget go down today. One instance was up and running the whole time but the other three all had login issues to the db (another issue as looking at the sql logs there was no failed login) but the /health response said everything was fine. Because the healthcheck was returning okay the servers with issues were never pulled out of the load balancer.

Actually, our email did not like us sending the files. I went ahead and logged a ticket [EDO-8787] so I could attach the files.

Alana, I have set answers to your questions and the package files to your email box. Thank you again for the support.

We are on Proget 2022.0.11 and we are chasing an issue trying to push a python package into Proget. It accepts the wheel, but is failing with a 400 Bad Request with the upload of the sdist tar.gz. The wheel shows in the Proget UI, but the sdist does not.

It shows an error of "Source distribution does not contain a PKG-INFO entry." I have checked the file and contents and the PKG-INFO does indeed exist the build seems to build things just fine. I just has an issue pushing to the internal repo.

Am I missing something here?