Just FYI, we found a solution that seemed to work - the packages that were failing all had a "quirky version number" warning in ProGet, and taking the ProGet-provided action of repackaging them with non-quirky version numbers seemed to resolve the issue.

Thanks for the quick reply @atripp! Regarding the condition potentially triggering the timeout where it seems like a request is getting hung up for more than a minute, does it seem from the corresponding ProGet log showing task cancelation like the hangup might be happening in ProGet, or do you feel that the intermediate network equipment is likely to be preventing ProGet from successfully providing a speedy response or not correctly relaying ProGet's response? If the hangup might be happening in ProGet, is there a revised, reasonable timeout (hopefully less than, say, 2.5 minutes) that you'd recommend instead of the curret 60-second timeout that would make it more likely that ProGet would eventually provide the desired response?

I could see a world where some results may take a while because NuGet package restore sends a large number of requests all at once and something somewhere (e.g. the C# thread pool or something similar in an intermediate network device) temporarily runs low on available resources and has to queue some requests to compensate, but in that case I feel like I'd see a pileup of varying longer response times. While some builds do have more than one package that times out, it's usually less than 10 out of around 300, and I don't see any middle ground between responses that come back in less than a second and responses that time out after sixty seconds.
 
I ask because I'd rather not go hunting down what seems like a potentially reasonable 60-second timeout enforced by intermediate network equipment if the only result is likely to be that it takes longer for builds to fail.

On a singleton instance of ProGet 2023.17 (build 4) we're running into some intermittent issues where retrieval of one or two NuGet packages will sometimes time out after 60 seconds with a 504 gateway timeout error, probably as a result of IIS timing out the request since this shows up in ProGet diagnostics as a canceled task. Other packages that don't fail to be retrieved during the same restore operation, though, typically take less than 1 second to retrieve, which makes me think that there's likely something getting stuck somewhere in the retrieval process on occasion.

We don't add new NuGet packages very often and run several builds each day, so I would expect all of the packages used by most builds to be cached by ProGet, and the package cited in this specific example is one that we've been referencing for a long time. We only started seeing this near the end of July, but unfortunately I'm not sure exactly what build we were on when it first started happening or what build we were on prior to that. Our builds reference a public feed backed by NuGet and a private feed containing a small number of internal packages.

Do you have any ideas around what might be causing this? If not, is there any additional diagnostic information that might be worth taking a look at to try to determine next steps to take? The stack trace from the diagnostic center reads to me like it could either be trying to add the package to the response or maybe trying to retrieve it from/verify it against nuget.org; could you clarify what ProGet is doing when the request is canceled?

Thanks for any assistance you may be able to offer!

Thanks @atripp, I appreciate the help! We're more concerned about reducing UI clutter than about space usage in this case, in particular because we're pushing data about pre-release builds in addition to production so we can catch issues ahead of releases but we don't need pre-release data to stick around. I'll keep in mind that SQL is another option for removing these and reclaiming space, though personally I'd prefer to stick with the API approach unless space becomes an issue.

I was looking for settings to be able to set up retention policies for releases in Reporting & SCA for ProGet 2023.6 without BuildMaster, and I didn't come across anything in Administration or find any documentation around that. I was planning to fall back on writing a tool for this myself, but SCA Projects & Releases API documentation doesn't appear to include an endpoint for deleting releases. If you would please let me know

1. whether it's possible to configure retention policies for ProGet Reporting & SCA releases at this time without using BuildMaster and where to find that setting if it is and
2. whether there's an API that I might have missed that would allow me to automate retention myself by writing tooling to delete ProGet releases as they become stale (maybe an update action that sets the active flag to false is equivalent to a delete?)

I'd appreciate it. Thanks!