Nevermind, we solved it. We had to also change the identity of the AppPool in IIS to point to our domain user. After changing it, it was working.

@dean-houston Oh okay. Weird. Is there also a path to access the status of each server in the cluster (similar to the one showed /administration/cluster/configure) ?

Hi @dean-houston, as you can see in the screenshot below, I am unable to access that page even from non-localhost endpoint. It just loads indefintely (no error or anything). Is it the port 33237 issue? Or could there be something else? It was working before (like 2-3 days ago).

How would I go about disabling the cluster in this case (since we have just done this on a test environment for now)?

@dean-houston @atripp , for some reason we are not able to load the ProGet Cluster Configuration page after clicking on "More Info":

The page tries to load indefintely as seen in the image.
However, you can see it still displays the status of the cluster. Does that mean there is something wrong with the load balancer?

Nevermind, we solved it. We had to also change the identity of the AppPool in IIS to point to our domain user. After changing it, it was working.

Hi @dean-houston, thank you for the reply. This is happening for every package consistently. After migrating to a shared storage space for all the package files, we are not able to access any of the files from ProGet. Here is the context of that error in more detail (if that helps):

How do you normally define UNC path (for network share) within ProGet settings?

Also, is there a difference between the user running the ProGet Web Service and IIS App pool? By default, the ProGet runs via a Network service, we changed the user to a domain user to run the service. That domain user is able to access the files via network share path on the ProGet machine:

Hi @atripp, we are currently working to migrate our ProGet to clustered solution. As part of the migration, we are first testing it out using a test DB and test files from our test ProGet instance (we cloned our test ProGet instance's drive).

Our team setup a shared storage space. We now run the ProGet service using a domain account that has access to the shared storage space. After modifying the path to "Storage.PackagesRootPath" to point to the shared storage space, we get the following error when trying to download a package:

Access to the path '\\<NAME_OF_SHARED_STORAGE_SPACE>\ProGet\ProGet\Packages\.nugetv2\<FEED_ID>\Amazon.CloudWatch.EMF\Amazon.CloudWatch.EMF.2.1.0.0.nupkg' is denied.

We made sure that the domain account has permissions for this shared storage. I can even access this path (via network file share UNC path) from the ProGet server using the domain account.

What could be the issue? How do you refer to network share paths in the settings (I just inserted the path as shown in the error, as it is in the field for "Storage.PackagesRootPath")?

Hi @atripp , UPDATE: We have decided to move forward with the multi-server approach. Our plan is to create fresh new servers and install the same version of ProGet on them. However, we would like to keep our existing single ProGet instance on until we migrate to the clustered solution.

Do you have any feed back for us in order to avoid possible road blocks and errors?

Also, do you recommend using Windows fileshare (as shared storage) between the multiple ProGet servers? How does ProGet handle writes to the same resource? Will it cause any issues/delay when multiple servers are trying to write to the same thing?

Do you also recommend multiple DB instances or is a single DB instance that multiple servers can connect to sufficient?

Hi @atripp, we've modified it to 100, however, we are still seeing time out issues during high peak. Also, we notice that the connection pool errors have went down but we face the following error still:

Connector <connector_name> Error: The operator has timed out. 

What did you mean by chained connectors? We have self connectors that point to "localhost".

Are there any other immediate measures we can take to avoid these timeouts? Otherwise, our only option would be to revert back to the previous version till we move to a clustered solution.

Note: There is somewhat of a faster recovery after changing to 100 concurrent requests.

Some added context:

We do not have a multi-server solution at the moment. It is a single ProGet instance hosted by IIS.

Also v6.0.20 had similar issues with slowness of ProGet on the UI side (when trying to access ProGet via web url, we got 500 and the website was unusable via UI). However, on v6.0.20 we did not get as many timeouts from API calls which seems odd.

Our performance degradation is very sporadic and has happened multiple times today (Note: We upgraded to 2024.16 on Monday 8 pm ET). We tried increasing our resources (memory and CPU) and also tried setting the "Web.ConcurrentRequestLimit" to 500 but it still has the same issue.

We generally use V2 API for a lot of these API calls and I know you suggested on this forum that we should move to v3 but that is not something we can do tomorrow right away (so is switching to clustered solution).

Are there any other recommendations you can make to remedy the situation to have immediate impact?

Reverting back to the old version is not ideal at all since the back up would take us to the state of the ProGet server (1 day ago) resulting in data loss.