Debian feed mirror Performance

dan.brown_0128

Is anybody else attempting to use Proget's debian feeds to mirror the official mirror for use in isolated environments?

I am curious what sort of performance others are seeing in that use case. Our prior tests found that mirroring via Proget was substantially slower than the official mirrors.

atripp

Hi Dan,

In general, a ProGet feed will be slower than a "real" Debian repository. The reason is that Debian repositories are just static file system like this:
http://ftp.us.debian.org/debian/dists/bookworm/

There is obviously a lot more overhead with each ProGet request, since index files are dynamically generated, involve connectors, need to be permission-checked, etc.

In addition, ProGet indexes the Debian indexes on-demand, which means downloading all of the Contents-*.gz files and indexing those. Like in here: http://ftp.us.debian.org/debian/dists/bookworm/main/

While these files are cached, they do need to be updated when the remote repository updates.

And note that each web node maintains its own local index cache, so you will see these long-running requests multiple times. Some organizations will periodically "warm up" the indexes periodically by just hitting the InRelease endpoint.

Hope that helps,
Alana

dan.brown_0128

To share with the community some things learned from a private support ticket -- There's currently a defect in the logic where if no packages were downloaded/uploaded to a debian type feed, the in-memory cache is never started. This leads to very long runtimes when running apt update since the response times for the /Release and /InRelease endpoints (from our experience) are at best 1 minute, at worst 10 minutes.

The workaround support gave was to manually download a package via http before attempting to use apt. From our experience, this doesn't persist overnight, so it would basically need done daily and truly is counterintuitive to having a package manager like apt.

stefan.hakansson_8938

Hi @dan-brown_0128 have you made any further progress on this? I'm seeing exactly the same behaviour, and it makes things like Ansible playbooks involving apt: update_cache: yes time out.

Any tip on how to improve the performance would be great (I have a default ProGet installation with default database and web server BTW).

Thanks,
Stefan

dan.brown_0128

@stefan-hakansson_8938 No, we haven't had any more luck and have put the debian mirror back on the backburner for the time being

stefan.hakansson_8938

@dan-brown_0128 thanks for the update. I'll ponder about what we should do and provide an update here if we find a solution involving ProGet as debian mirror with a decent performance.

stevedennis

@stefan-hakansson_8938 as you noticed, ProGet's Debian connectors are not currently designed to handle the gigantic, operating-system mirrors very well. This is because they are always refreshed "on demand" - which is what you want for CI/CD workflows.

It's not great for public repository mirroring, however. In Q4 or later, we will explore adding an option to do periodic updates.

stefan.hakansson_8938

@stevedennis thank you for the update and insights. Looking forward to what Q4 (or later) will bring!

stno_9153

This post is deleted!