This is happening across all Maven feeds even private ones

An error occurred in the web application: Object reference not set to an instance of an object.

URL: http://proget.wtg.zone/feeds/Maven-WTG/com.cargowise.eservices/Maven Health Check API/1.0.0
Referrer: https://proget.wtg.zone/packages?FeedId=20&Cache=all
User: SW
User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0
Stack trace: at Inedo.ProGet.Feeds.Maven.MavenFeed.Inedo.ProGet.Feeds.IBrowsablePackageFeed.ListPackageVersionsAsync(BrowsablePackageId id)
at Inedo.ProGet.WebApplication.Pages.Packages.PackagePageBase.CreateChildControlsAsync()
at Inedo.ProGet.WebApplication.Pages.ProGetSimplePage.InitializeAsync()
at Inedo.Web.PageFree.SimplePageBase.ExecutePageLifeCycleAsync()
at Inedo.Web.PageFree.SimplePageBase.ProcessRequestAsync(AhHttpContext context)
at Inedo.Web.AhWebMiddleware.InvokeAsync(HttpContext context)

We are also getting the following error when trying to access packages on the Feed

An error occurred in the web application: Object reference not set to an instance of an object.

URL: http://proget.wtg.zone/feeds/Maven/nekohtml/Neko HTML/1.9.6.2
Referrer: https://proget.wtg.zone/packages?FeedId=3&Cache=cached&skip=0&take=1000
User: SW
User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0
Stack trace: at Inedo.ProGet.Feeds.Maven.MavenFeed.Inedo.ProGet.Feeds.IBrowsablePackageFeed.ListPackageVersionsAsync(BrowsablePackageId id)
at Inedo.ProGet.WebApplication.Pages.Packages.PackagePageBase.CreateChildControlsAsync()
at Inedo.ProGet.WebApplication.Pages.ProGetSimplePage.InitializeAsync()
at Inedo.Web.PageFree.SimplePageBase.ExecutePageLifeCycleAsync()
at Inedo.Web.PageFree.SimplePageBase.ProcessRequestAsync(AhHttpContext context)
at Inedo.Web.AhWebMiddleware.InvokeAsync(HttpContext context)

We have an issue where Maven downloads from the public repository (https://repo1.maven.org/maven2) are taking forever. It is timing out.

Hi Support,

We have a public Maven feed for which we cannot see the artefacts. We left it for 20 minutes, but it showed loading artefacts.

Regards Scott

Hi Support,

On our new installs of ProGet, we noticed that the IIS Application Pool Managed Pipeline Mode is classic. Is there any reason these cannot be integrated?

Regards Scott

A possible solution to this is to use Extended Properties in SQL. That way you can check the version of the files in the upgrade and not re-create them when they have already been updated?

@atripp We had the same error when upgrading each node on a high availability install. Does the upgrade routine delete and recreate the stored procedures for each node upgrade? It felt like there was no check of a version for each node upgrade.

When you say Web Server Windows Service, I think you should put the account against the ProGetAppPool Identity. We are running as an integrated SQL connection, and the account is a DB Owner. Both the APP Pool and ProGet Service run as this account. We added the following to the connection string Persist Security Info=False;Integrated Security=true;

Yes, Using SELECT * FROM FeedPackageVersions_ExtendedWithMaven. I have found several packages with null Package_Names across the Maven feeds. The ones for the Maven-WTG can be seen here.

So there are a few in there. We used the bulk import process to bring these across. The Maven feed is a public cache feed, and these were all downloaded. Latest version of ProGet

Hi Support,

We have a problem with the packages option in ProGet. We are getting the following error when selected.

We have narrowed it down to one feed, the Maven-WTG feed. The others are working without issues.

There is an error in the diagnostic centre.

Message: Value cannot be null. (Parameter 'stringToEscape')

Details:

System.ArgumentNullException: Value cannot be null. (Parameter 'stringToEscape')
at System.UriHelper.EscapeString(String stringToEscape, Boolean checkExistingEscaped, ReadOnlySpan`1 unreserved, Char forceEscape1, Char forceEscape2)
at Inedo.ProGet.WebApplication.Pages.Packages.PackageOverviewPage.BuildUrl(FeedId feedId, String group, String name, String version)
at Inedo.ProGet.WebApplication.Controls.PackageLink.GetPackageUrl(Int32 feedId, String packageId, String groupId, String version)
at Inedo.ProGet.WebApplication.Controls.PackageLink..ctor(FeedPackageVersions_Extended p)
at Inedo.ProGet.WebApplication.Pages.Packages.ListLocalPackagesPage.<>c__DisplayClass18_1.<<CreateChildControls>g__enumerat|6>d.MoveNext()
at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
at Inedo.Web.Controls.AhTable.Inedo.Web.Controls.ISimpleCallbackHandler.ProcessRequestAsync(AhHttpContext context, String eventName, String eventArgument)

Excellent, we have just moved to using integrated security with the SQL server. The account was configured as DB owner. Will test the next upgrade when available.

Thank you for the information. I will investigate the virus scanner

Hi Support,

We have created a cluster of 3 ProGet nodes. When looking at the cluster management page, we get a message about "Cannot connect to service messenger." This changes on the servers as the HA proxy redirects to another node.

We have confirmed that port 33237 is accessible across the server, so I am not sure why we are getting this error. Also, if this is a problem?

Regards Scott

Hi Support,

We have a new high availability install and are testing the feed setup. We are testing the feed and hitting a number of errors that look like it may have something to do with the shared storage. We are not getting these errors on our NuGet and Maven feeds.

An error occurred processing a GET request to http://proget.wtg.zone/npm/Registry/regexpu-core/-/regexpu-core-4.2.0.tgz: The process cannot access the file '\SMBShare\Proget\Packages.npm\F9\regexpu-core\4.2.0\package.tgz' because it is being used by another process.

An error occurred processing a GET request to http://proget.wtg.zone/npm/Registry/@vueuse/core/-/core-10.4.1.tgz: Uncommittable transaction is detected at the end of the batch. The transaction is rolled back.

The builds continue to work even though we are getting this error but they are filling up the logs.

Regards Scott

Hi Support,

We have set up Proget with a high availability structure. It has three Windows nodes running IIS, a single SQL Enterprise server, and SMB shared storage. When installing, we ran Inedo Hub on each of the node servers, installing IIS and ProGet on them. After installing the first node, we encountered errors with the other nodes; we could not find a way to install them and point to the existing database. We had to install using a new database name and update the connection string afterwards. Is there a way to easily install extra nodes that we have missed?

Also, what is the process for upgrading ProGet in a HA setup? We have been unable to find instructions on upgrades for HA and are worried that updating one node will create the same problems we had with the install. Can we please get an indication of the correct install process and upgrade routine?

Regards Scott

Hi Atripp,

We have received some scripts from Alex Papadimoulis. We started running these yesterday and the SQL blocking has so far disappeared and the Dashboards_GetLargestPackages now has no issues. Looks like Brett Shearer and Alex have a solution for both problems. Our load is still an issue at peak times but features are now working fast. Will keep monitoring and get back to Alex confirming the outcomes of the new stored procedures next week.

Thank you

Hi Everyone,

We have upgraded from ProGet 2022.30 to 2023.28. After the upgrade when selecting the Storage And Retention option on the feed we get a timeout. Other feeds have no issue but when selecting the one for the public NuGet Gallery proxy we get the following error.

The logs contain this:

URL: https://proget.wtg.zone/feed/manage/storage?feedId=1
Referrer: https://proget.wtg.zone/feed/manage?feedId=1
User: SW
User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 Edg/120.0.0.0
Stack trace: at Microsoft.Data.SqlClient.SqlCommand.<>c.<ExecuteDbDataReaderAsync>b__209_0(Task`1 result)
at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
at Inedo.Data.DatabaseContext.DbResult.CreateAsync(DbCommand command, DatabaseContext context, DateTimeOffset startTime, Stopwatch stopwatch)
at Inedo.Data.DatabaseContext.ExecuteInternalAsync(String storedProcName, GenericDbParameter[] parameters)
at Inedo.Data.SqlServerDatabaseContext.ExecuteInternalAsync(String storedProcName, GenericDbParameter[] parameters)
at Inedo.Data.DatabaseContext.ExecuteTableAsync[TRow](String storedProcName, GenericDbParameter[] parameters)
at Inedo.ProGet.WebApplication.Pages.Feeds.Manage.ManageFeedStoragePage.BuildLargestPackages(StyledTable largestPackagesTable)
at Inedo.ProGet.WebApplication.Pages.Feeds.Manage.ManageFeedStoragePage.AddContentControlsAsync(ContentContainer content)
at Inedo.ProGet.WebApplication.Pages.Feeds.Manage.ManageFeedPageBase.CreateChildControlsAsync()
at Inedo.ProGet.WebApplication.Pages.ProGetSimplePage.InitializeAsync()
at Inedo.Web.PageFree.SimplePageBase.ExecutePageLifeCycleAsync()
at Inedo.Web.PageFree.SimplePageBase.ProcessRequestAsync(AhHttpContext context)
at Inedo.Web.AhWebMiddleware.InvokeAsync(HttpContext context)

We are noticing the following query having a duration of anywhere from 12-20 seconds. Our Server is incredibly slow.

SELECT @PackageName_Id = [PackageName_Id]
FROM [PackageNameIds] WITH (UPDLOCK, SERIALIZABLE) 
WHERE [PackageType_Name] = @PackageType_Name
		       AND (([PackageGroup_Name_Lower] IS NULL AND @PackageGroup_Name IS NULL) OR ([PackageGroup_Name_Lower] = LOWER(@PackageGroup_Name)))
		       AND [Package_Name_Lower] = LOWER(@Package_Name)

Is there a reason why the Create_Indicator is always defaulting to 'Y' PackageNameIds_GetOrCreatePackageNameId. Have you considered that doing the commit with UPDLOCK and SERIALIZABLE could be causing issues with AlwaysOn and sycnchronsist commits going to the secondary? It appears everything is being blocked waiting for the secondary to update and we may not need to update at all.

Instead of doing pessimistic locking in the stored procedure it could be changed to do optimistic locking.

GET Id from table
if ID is null
	insert row
	if error
		Get ID from table
	else
		Get ID from whatever you inserted
	end
end
return ID

Hi Inedo,

Had an issue with the feed upgrade failing due to old packages with version issues. Was able to resolve all of these and complete the upgrade process. The server is up and running and responding to package requests across feeds are working. The issue is the event viewer we are getting .Net Runtime errors for 2 feeds npm and nuget cache feeds.

Category: Microsoft.AspNetCore.Server.IIS.Core.IISHttpServer
EventId: 2
SpanId: f132db9462672ba5
TraceId: 72864a074b90e5fac4c87f08e7c5ce94
ParentId: 0000000000000000
RequestId: 80000a9e-000e-c900-b63f-84710c7967bb
RequestPath: /npm/Registry/npm

Connection ID "14483576463363672733", Request ID "80000a9e-000e-c900-b63f-84710c7967bb": An unhandled exception was thrown by the application.

Exception: 
System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at Inedo.Web.AhHttpResponse.CompleteAsync()
   at Inedo.Web.AhWebMiddleware.InvokeAsync(HttpContext context)
   at Inedo.Web.AhWebMiddleware.InvokeAsync(HttpContext context)
   at Inedo.Web.AhWebHost.<>c.<<Configure>b__18_0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContextOfT`1.ProcessRequestAsync()

Category: Microsoft.AspNetCore.Server.IIS.Core.IISHttpServer
EventId: 2
SpanId: ecc2892673882cd1
TraceId: d46b306d137da872e9d90b192c3cdc99
ParentId: 0000000000000000
RequestId: 800007d4-000a-e900-b63f-84710c7967bb
RequestPath: /nuget/Gallery/v3/flatcontainer/microsoft.net.workload.mono.toolchain.net6.manifest-8.0.100.msi.x64/index.json

Connection ID "16789419455397496785", Request ID "800007d4-000a-e900-b63f-84710c7967bb": An unhandled exception was thrown by the application.

Exception: 
System.OperationCanceledException: The operation was canceled.
   at System.Threading.CancellationToken.ThrowOperationCanceledException()

The Web.ConcurrentRequestLimit is set to 0. We only started to see these errors after the upgrade. Any idea what it could be?

Regards Scott

Thank you atripp, confirmation of everything we thought was the problem and working on it now.