@dean-houston Oh okay. Weird. Is there also a path to access the status of each server in the cluster (similar to the one showed /administration/cluster/configure) ?
Posts made by sneh.patel_0294
-
RE: Performance Issues after upgrading ProGet to v2024.16 from v6.0.20
-
RE: Performance Issues after upgrading ProGet to v2024.16 from v6.0.20
Hi @dean-houston, as you can see in the screenshot below, I am unable to access that page even from non-localhost endpoint. It just loads indefintely (no error or anything). Is it the port 33237 issue? Or could there be something else? It was working before (like 2-3 days ago).
How would I go about disabling the cluster in this case (since we have just done this on a test environment for now)?
-
RE: Performance Issues after upgrading ProGet to v2024.16 from v6.0.20
@dean-houston @atripp , for some reason we are not able to load the ProGet Cluster Configuration page after clicking on "More Info":
The page tries to load indefintely as seen in the image.
However, you can see it still displays the status of the cluster. Does that mean there is something wrong with the load balancer? -
RE: Performance Issues after upgrading ProGet to v2024.16 from v6.0.20
Nevermind, we solved it. We had to also change the identity of the AppPool in IIS to point to our domain user. After changing it, it was working.
-
RE: Performance Issues after upgrading ProGet to v2024.16 from v6.0.20
Hi @dean-houston, thank you for the reply. This is happening for every package consistently. After migrating to a shared storage space for all the package files, we are not able to access any of the files from ProGet. Here is the context of that error in more detail (if that helps):
How do you normally define UNC path (for network share) within ProGet settings?
Also, is there a difference between the user running the ProGet Web Service and IIS App pool? By default, the ProGet runs via a Network service, we changed the user to a domain user to run the service. That domain user is able to access the files via network share path on the ProGet machine:
-
RE: Performance Issues after upgrading ProGet to v2024.16 from v6.0.20
Hi @atripp, we are currently working to migrate our ProGet to clustered solution. As part of the migration, we are first testing it out using a test DB and test files from our test ProGet instance (we cloned our test ProGet instance's drive).
Our team setup a shared storage space. We now run the ProGet service using a domain account that has access to the shared storage space. After modifying the path to "Storage.PackagesRootPath" to point to the shared storage space, we get the following error when trying to download a package:
Access to the path '\\<NAME_OF_SHARED_STORAGE_SPACE>\ProGet\ProGet\Packages\.nugetv2\<FEED_ID>\Amazon.CloudWatch.EMF\Amazon.CloudWatch.EMF.2.1.0.0.nupkg' is denied.
We made sure that the domain account has permissions for this shared storage. I can even access this path (via network file share UNC path) from the ProGet server using the domain account.
What could be the issue? How do you refer to network share paths in the settings (I just inserted the path as shown in the error, as it is in the field for "Storage.PackagesRootPath")?
-
RE: Performance Issues after upgrading ProGet to v2024.16 from v6.0.20
Hi @atripp , UPDATE: We have decided to move forward with the multi-server approach. Our plan is to create fresh new servers and install the same version of ProGet on them. However, we would like to keep our existing single ProGet instance on until we migrate to the clustered solution.
Do you have any feed back for us in order to avoid possible road blocks and errors?
Also, do you recommend using Windows fileshare (as shared storage) between the multiple ProGet servers? How does ProGet handle writes to the same resource? Will it cause any issues/delay when multiple servers are trying to write to the same thing?
Do you also recommend multiple DB instances or is a single DB instance that multiple servers can connect to sufficient?
-
RE: Performance Issues after upgrading ProGet to v2024.16 from v6.0.20
Hi @atripp, we've modified it to 100, however, we are still seeing time out issues during high peak. Also, we notice that the connection pool errors have went down but we face the following error still:
Connector <connector_name> Error: The operator has timed out.
What did you mean by chained connectors? We have self connectors that point to "localhost".
Are there any other immediate measures we can take to avoid these timeouts? Otherwise, our only option would be to revert back to the previous version till we move to a clustered solution.
Note: There is somewhat of a faster recovery after changing to 100 concurrent requests.
-
RE: Performance Issues after upgrading ProGet to v2024.16 from v6.0.20
Some added context:
We do not have a multi-server solution at the moment. It is a single ProGet instance hosted by IIS.
Also v6.0.20 had similar issues with slowness of ProGet on the UI side (when trying to access ProGet via web url, we got 500 and the website was unusable via UI). However, on v6.0.20 we did not get as many timeouts from API calls which seems odd.
Our performance degradation is very sporadic and has happened multiple times today (Note: We upgraded to 2024.16 on Monday 8 pm ET). We tried increasing our resources (memory and CPU) and also tried setting the "Web.ConcurrentRequestLimit" to 500 but it still has the same issue.
We generally use V2 API for a lot of these API calls and I know you suggested on this forum that we should move to v3 but that is not something we can do tomorrow right away (so is switching to clustered solution).
Are there any other recommendations you can make to remedy the situation to have immediate impact?
Reverting back to the old version is not ideal at all since the back up would take us to the state of the ProGet server (1 day ago) resulting in data loss.
-
Performance Issues after upgrading ProGet to v2024.16 from v6.0.20
We are experiencing severe performance issues and a lot of timeouts for ProGet. We recently upgraded to ProGet v2024.16 from v6.0.20. We are seeing the following error in the logs:
An error occurred processing a GET request to "https://<proget_endpoint>": Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached. System.InvalidOperationException: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
We have New Relic monitoring installed on our ProGet server and here is the data:
Response times are VERY high:
Spikes in CPU usage that coincide with the period of performance degradation:
Spikes in memory usage that coincide with the period of performance degradation:
Huge spike in GC time during the period of performance degradation:
Notice in the charts for response time, CPU usage, and memory usage, there are moments when New Relic had no data. This probably suggests that ProGet was down during those moments. What could be causing this?
Could you please help us with this issue?
-
RE: GET Request "lt" operator not working for a string in ProGet v2024.16
@atripp said in GET Request "lt" operator not working for a string in ProGet v2024.16:
ou can also use the NuGet v3 AP
Thank you for your response. Could you please point me to the NuGet v3 API documentation you are referring to? Also, how would I use the Packages API to do what the GET request is doing? The documentation for the Packages API is a bit confusing.
-
GET Request "lt" operator not working for a string in ProGet v2024.16
Hi, we are currently testing the upgrade from version ProGet 6.0.20 to 2024.16 on a test ProGet instance. One of the tests resulted in an error relating to a simple GET request:
An error occurred processing a GET request to https://test-endpoint/nuget/auto-int-cps/Packages()?$filter=Id eq 'CPS.Regression' and Version lt '9.1' and IsPrerelease eq 'False'&$top=1&$orderby=Version desc: The binary operator LessThan is not defined for the types 'System.String' and 'System.String'
Note: Our test endpoint points to our test ProGet instance. If I change the test endpoint to our prod endpoint (pointing to our prod instance sitting on version 6.0.20), the same GET request works.
Did the filter stop working in the newer version of ProGet? If so, what is the workaround?