ProGet feeds EXTREMELY slow when accessed by Octopus Deploy

We're new to ProGet. We've only recently started using Octopus Deploy. One project has 43 NuGet packages. These were kept on a simple file server (nothing special). Our next phase is to implement more secure package server and ProGet was chosen.

Using TeamCity to build the NuGet packages, we have no problem in getting these on the ProGet server.

However, Octopus Deploy experiences severe delays when trying to deploy multiple packages, to the point where the Octopus component on the target server (the Tentacle) times out and terminates the connection, resulting in a deployment failure.

Testing with a single package from the ProGet server, it takes Octopus 10-15 seconds to acquire the package. If I test using 2 packages, it takes 120 seconds.

Note, that over time, we're seeing some improvement. At first, we couldn't do 3 or more packages before it failed. Now we seem to get 4 or more, and the time to acquire is getting shorter, as if there is some caching going on.

Is there something we need to do to get more caching setup?
is there something we need to do to get better performance overall?

Note: I can do a simple SQL script and interrogate the database in a split second to find the package (but that's not downloading it). However, in Octopus, we can "test" the external feed by submitting a package name, and it will return the available version. this takes 10-15 seconds.

I am asking both Inedo and Octopus Deploy for help with this.

the performance we have right now is not acceptable. Please help or provide information on how we can get this better.

Product: ProGet
Version: 3.8.6

The first thing you should do is upgrade to the latest version... there have been dozens of performance improvements in the past 2 years that might resolve your issue.

Do you have documentation detailing the performance improvements/fixes that may help us?

There are simply too many to enumerate, but you're welcome to look through complete list of changes here: http://inedo.com/proget/versions.

Upgrading takes minutes, so I would just start there.

Do note that NuGet v2 (which Octopus uses) is a very chatty and inefficient protocol (this is why so many customers use load balanced installations), so with this set-up, you may never find acceptable performance... especially once you go to scale.

On the topic of upgrading and improving process, I would take a look at Deployment Automation vs Infrastructure as Code. We're seeing a lot of customers going this route, because it's a lot faster and allows for scaling so much more easily.

Hello Alana, thanks for the feedback.

The good news is we're making progress. In Octopus, we added the system variable settings:
Octopus.Acquire.MaxParallelism = 2
Controls the number of package acquisitions that will be allowed to run concurrently.
Octopus.Acquire.DeltaCompressionEnabled = true
Toggle whether delta compression is enabled when sending packages to targets.
Octopus.Deployment.ForcePackageDownload = false
If true, the package will be freshly downloaded from the feed/repository regardless of whether it is already present on the endpoint (Boolean)

These settings allowed us to complete the deployment of 43 packages.

Note: when using a file server repository, this was done in 4 minutes. Using ProGet we at 33 minutes, so we still have work to do.

You mentioned NuGet V2 being "chatty". If we had good performance using just a file server, is NuGet v2 actually a factor? The only difference within Octopus is which external feed is selected for the package.

I will look at upgrading ProGet, as well as the link.

Well, using a file server does not use NuGet API... it's just files on disk. But of course, using a file server instead of a repository for your packages is like using a share drive for your source code. Sure, it works, and maybe it's faster and simpler sometimes, but the benefits to source control vastly outweigh the convenience.

Wanted to follow up. we will be upgrading ProGet eventually. The thing is, our sister org in Melbourne AU is using the same version of ProGet (3.8.6).
I can test just using NuGet.EXE from a command prompt, to list the packages from a feed. When I try to list from ours, it takes nearly 25 seconds to reply with about 1/2 of the 43 packages on the feed, and other 25 seconds to get the rest.
Using the same NuGet.EXE list sample, but instead choose a feed from our AU server, the feed has over 680 packages, is on the other side of our planet, and starts returning package info in about 7 seconds, spewing more every 1-2 seconds, until a list of all 680+ packages is provided in just under 30 seconds.
Both sites are using ProGet 3.8.6.
We use IIS.
I can use SQL Manager and query the database and in a tiny fraction of a second we have all the data available.

It certainly seems like the IIS interface to ProGet may be the culprit here. Is there any particular setting (or settngs) that may impact performance like this?

It seems like the problem is caused outside of ProGet.

So, i would investigate Windows integrated Authentication (if you have it enabled, disable it as a test). Also look for other "random" IIS settings that may have inadvertently been changed. It's also possible to be networking equipment (bad router, switch, etc).