Hi,

I've been trying to set up ProGet on an Azure Kubernetes Cluster, which is going well so far but I found a small glitch where the admin landing page "https://<server>/administration" fails with a timeout if there are a very large number of records in the LogMessages table.

In my case it was about 300,000 messages due to an unrelated AKS configuration error I had with our container instance.

Deleting all the records from the LogMessages table using SQL Server Management Studio seemed to resolve the issue and the admin page loaded instantly afterwards.

Just posting in case this helps someone else in a similar situation...

Cheers,

M

Hello,

Just posting this here for visibility really as I've got a viable workaround...

I've been working on some housekeeping scripts to delete a large number of legacy package versions from a private NuGet feed (hosted on ProGet running in a container on an Azure Kubernetes with an Azure SQL Database in case the details matter) using the ProGet Delete Package Endpoint.

Due to the number of package versions involved (10's of thousands) I'm running the deletes via the API in parallel from multiple threads (specifically, I'm using PowerShell's foreach-object -parallel and Invoke-RestMethod) to delete specific versions of packages.

When this script is running I frequently see failed requests like the following in the ProGet Diagnostic Log while it's actively deleting multiple package versions in parallel:

Unhandled exception processing http://[myserver]/api/packages/[myfeed]/delete?name=[mypackage]&version=[myversion]
Details:Microsoft.Data.SqlClient.SqlException (0x80131904): 1205`13`51`FeedPackageVersions_DeletePackageVersion`36`Transaction (Process ID 218) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
Transaction count after EXECUTE indicates a mismatching number of BEGIN and COMMIT statements. Previous count = 1, current count = 0.

...snip ...

at Inedo.ProGet.Feeds.StandardPackageFeed`3.DeletePackageFromDatabase(PackageVersionId id, Context db) in C:\Users\builds\AppData\Local\Temp\InedoAgent\BuildMaster\192.168.44.60\Temp\_E400384\Src\ProGetCoreEx\Feeds\StandardPackageFeed.cs:line 760

and matching client-side errors like:

Response         : StatusCode: 500, ReasonPhrase: 'Internal Server Error', Version: 1.1, Content: System.Net.Http.HttpConnectionResponseContent, Headers:
                   {
                     Date: Thu, 18 Jan 2024 09:24:04 GMT
                     Connection: keep-alive
                     Cache-Control: private
                     X-ProGet-Version: 23.0.24.3
                     Strict-Transport-Security: max-age=15724800; includeSubDomains
                     Content-Type: text/plain
                     Content-Length: 337
                   }
HttpRequestError : Unknown
StatusCode       : InternalServerError
TargetSite       : Void ThrowTerminatingError(System.Management.Automation.ErrorRecord)
Message          : Response status code does not indicate success: 500 (Internal Server Error).
Data             : {}
InnerException   : 
HelpLink         : 
Source           : System.Management.Automation
HResult          : -2146233088
StackTrace       :    at System.Management.Automation.MshCommandRuntime.ThrowTerminatingError(ErrorRecord errorRecord)

If I run the diagnostic steps in Analyze and prevent deadlocks in Azure SQL Database it tells me that the deadlock is being caused by two instances of the following query running at the same time:

SELECT @FPCT = (
SELECT COUNT(*)
FROM [FeedPackages] FP WITH (UPDLOCK, SERIALIZABLE)
LEFT JOIN [PackageVersionIds] PVI WITH (UPDLOCK,SERIALIZABLE) ON
PVI.[PackageName_Id] = FP.[PackageName_Id]
LEFT JOIN [FeedPackageVersions] FPV WITH (UPDLOCK,SERIALIZABLE) ON
PVI.[PackageVersion_Id] = FPV.[PackageVersion_Id]
WHERE FP.[Feed_Id] = @Feed_Id AND FP.[PackageName_Id] = @PackageName_Id

In terms of impact, my script has cleaned up about 99% of the historic package versions that we need to delete, so I've resolved the issue by running it again single-threaded to delete the remaining few package versions (using foreach-object -throttlelimit 1 -parallel) - this worked fine so it's not a blocker, but it would be nice to be able to run the entire script in one go using multiple threads for improved performance.

Cheers,

M

Will do - happy to be a guinea pig :-)

I've found one small hiccup already I think - our Azure Active Directory instance has enforced MFA on accounts, so I won't be able to use Windows Authentication for ProGet to connect to SQL Server and I've been advised to use a Service Principal instead, but System.Data.SqlClient doesn't support authenticating to SQL Server with a service principal:

# build the connection string
PS C:\temp> $connectionString = ([ordered] @{
    "Server"         = "my-sql-server.database.windows.net"
    "Authentication" = "Active Directory Service Principal"
    "Encrypt"        = "True"
    "Database"       = "my-sql-database"
    "User Id"        = "00000000-0000-0000-00000000000"
    "Password"       = "<secret-generated-by-azure>"
}).GetEnumerator() | foreach-object { $_.Name + "=" + $_.Value } | join-string -Separator ";"

# show the connection string
PS C:\temp> $connectionString
Server=my-azure-database.database.windows.net;Authentication=Active Directory Service Principal;Encrypt=True;Database=my-proget-database;User Id=00000000-0000-0000-0000-000000000000;Password=<secret-generated-by-azure>

PS C:\temp>.\inedosql.exe update . --connection-string=$connectionString

Unhandled Exception: System.ArgumentException: Invalid value for key 'authentication'.
   at System.Data.Common.DbConnectionStringBuilderUtil.ConvertToAuthenticationType(String keyword, Object value)
   at System.Data.SqlClient.SqlConnectionString.ConvertValueToAuthenticationType()
   at System.Data.SqlClient.SqlConnectionString..ctor(String connectionString)
   at System.Data.SqlClient.SqlConnectionFactory.CreateConnectionOptions(String connectionString, DbConnectionOptions previous)
   at System.Data.ProviderBase.DbConnectionFactory.GetConnectionPoolGroup(DbConnectionPoolKey key, DbConnectionPoolGroupOptions poolOptions, DbConnectionOptions& userConnectionOptions)
   at System.Data.SqlClient.SqlConnection.ConnectionString_Set(DbConnectionPoolKey key)
   at System.Data.SqlClient.SqlConnection.set_ConnectionString(String value)
   at System.Data.SqlClient.SqlConnection..ctor(String connectionString, SqlCredential credential)
   at Inedo.DbUpdater.SqlServer.SqlServerDatabaseConnection.GetConnection()
   at Inedo.DbUpdater.SqlServer.SqlServerDatabaseConnection.ExecuteTable[TResult](String query, Func`2 adapter, SqlTransactiontransaction, SqlParameter[] args)
   at Inedo.DbUpdater.SqlServer.SqlServerDatabaseConnection.GetChangeScriptVersion(SqlTransaction transaction)
   at Inedo.DbUpdater.SqlServer.SqlServerDatabaseConnection.GetState()
   at Inedo.DbUpdater.Program.Update(String scriptPath, String connectionString, Boolean force)
   at Inedo.DbUpdater.Program.Run(ArgList args)
   at Inedo.DbUpdater.Program.Main(String[] args)

See System.ArgumentException: Invalid value for key 'authentication' above.

The answer is apparently to migrate to the Microsoft.Data.SqlClient package which supports additional Azure-based authentication methods - that's obviously quite a big change though but I figured I'd mention it so you were aware...

See https://learn.microsoft.com/en-us/sql/connect/ado-net/sql/azure-active-directory-authentication?view=sql-server-ver16#using-service-principal-authentication for more details.

Here's the equivalent login from SQL Server Management Studio, which works with the credentials I've put in my connection string above:

In the meantime I'll follow up with our secops team to see if I can get an exemption from the MFA policy for a new Azure AD account...

Hi @atripp - thanks for the reply - I've only just seen this after the Christmas holidays :-)

Just to clarify, we only plan to run a single instance of a ProGet container but it'll be running on an AKS instance - the "cluster" I was referring to is the underlying Kubernetes platform as opposed to, say, a cluster of HA ProGet server containers, so I think the Basic edition should be fine.

I understand what you're saying about supporting your product vs supporting the infrastructure it runs on - we've got in-house SMEs for Kubernetes and Azure so we can support the infrastructure ourselves and it sounds like you're prepared to provide product support for containerised instances running in Azure / AKS. For a contrived example let's say we're having problems with AD integration - if you're able to help troubleshoot error messages and identify that ProGet is unable to talk to a domain controller we'd certainly be capable of investigating the underlying Azure / AKS configuration ourselves for issues with, e.g. vnets, subnets, nsgs, etc.

I'm going to try to spin up a PoC using a trial license over the next week or so, and if that meets our requirements (I'm assuming it will from your documentation) then I expect our procurement / service delivery guys will be in touch to get a formal statement re support, but I'll leave that to them to organise :-).

Thanks again,

Mike

Hello,

For the last few years we've been happily running a Free instance of ProGet on an Azure Windows virtual machine.

We're looking to upgrade to a Basic license early in the new year to unlock some of the additional features, and as part of that we'd like to migrate to running as a container on an Azure Kubernetes Cluster instead of a Windows vm.

Looking at other tickets in this forum, some of your customers are already doing that but I wanted to check what the official line is from your support team? We have a number of people in-house who administer and maintain the base cluster, but we want to make sure we stay on the official ProGet support matrix as well.

Could you confirm whether running the ProGet image on a Kubernetes cluster is a supported configuration? Or is support only available for Docker-based containers?

Thanks,

Mike