Debian feed mirror Performance

[dan.brown_0128]

Is anybody else attempting to use Proget's debian feeds to mirror the official mirror for use in isolated environments?

I am curious what sort of performance others are seeing in that use case. Our prior tests found that mirroring via Proget was substantially slower than the official mirrors.

[atripp]

Hi Dan,

In general, a ProGet feed will be slower than a "real" Debian repository. The reason is that Debian repositories are just static file system like this:
http://ftp.us.debian.org/debian/dists/bookworm/

There is obviously a lot more overhead with each ProGet request, since index files are dynamically generated, involve connectors, need to be permission-checked, etc.

In addition, ProGet indexes the Debian indexes on-demand, which means downloading all of the Contents-*.gz files and indexing those. Like in here: http://ftp.us.debian.org/debian/dists/bookworm/main/

While these files are cached, they do need to be updated when the remote repository updates.

And note that each web node maintains its own local index cache, so you will see these long-running requests multiple times. Some organizations will periodically "warm up" the indexes periodically by just hitting the InRelease endpoint.

Hope that helps,
Alana

[dan.brown_0128]

To share with the community some things learned from a private support ticket -- There's currently a defect in the logic where if no packages were downloaded/uploaded to a debian type feed, the in-memory cache is never started. This leads to very long runtimes when running apt update since the response times for the /Release and /InRelease endpoints (from our experience) are at best 1 minute, at worst 10 minutes.

The workaround support gave was to manually download a package via http before attempting to use apt. From our experience, this doesn't persist overnight, so it would basically need done daily and truly is counterintuitive to having a package manager like apt.