Incomplete proget debian connector local index file for ubuntu noble-backports dist

[dimas]

I am trying to create debian connector to the ubuntu noble repositories (noble, noble-updates, noble-security and noble-backports). All is going pretty smooth, but one, the noble-backports dist never produce a complete local index file.

This is the connector configuration:

The resulting local index file (http://host.docker.internal/debian/ubuntu/dists/noble-backports/Release) always omits the restricted and universe components.

Suite: noble-backports
Codename: noble-backports
Date: Tue, 27 May 2025 10:10:55 UTC
Architectures: amd64 i386
Components: main universe
MD5Sum:
 870b00fd37986f702efacd24a1ac1587           253683 main/binary-amd64/Packages
 594b4e7cc78fe60b0e8fd65d4883fe1c            48470 main/binary-amd64/Packages.gz
 985068beb16bd85eb70cad4c948f92ad           199331 main/binary-i386/Packages
 f756fbb5c259d3aec79acfd8027b8036            39299 main/binary-i386/Packages.gz
 7d00419b8b0e13fa6def0b6bd07a35fd           132133 universe/binary-amd64/Packages
 7670c8af86a1d5c6bf33f9dc31125705            31953 universe/binary-amd64/Packages.gz
 1e09e22a87efe2711f9c8500a3ae1707            63809 universe/binary-i386/Packages
 aaab9d428881ccf5196952baf811a34d            16063 universe/binary-i386/Packages.gz
SHA1:
 905488d65751f1c9e41c276039d6df5cf0243f39           253683 main/binary-amd64/Packages
 6b04e30059820a4641c724c56fcbddce1e0504cb            48470 main/binary-amd64/Packages.gz
 5748335bdf5b57b849915fb8fd15079645e1b421           199331 main/binary-i386/Packages
 636e7ea621ac09847f88287cc2e0fcb3d37b5ff9            39299 main/binary-i386/Packages.gz
 2a669eff667ae4d456d3bdde7a928f31c378f9db           132133 universe/binary-amd64/Packages
 05af0407b7c860f75efd8c3f66a9f562287e1f36            31953 universe/binary-amd64/Packages.gz
 4196f2c27cf3c029601462fefc082b2a1264d337            63809 universe/binary-i386/Packages
 b3179ba9868d6c75ed5087540692084fe45f5c56            16063 universe/binary-i386/Packages.gz
SHA256:
 8fc457bc209518eae85c6245db18b4d5b5b5782198f89d8471fddc12c7c300f9           253683 main/binary-amd64/Packages
 a362b90692ff0f68c7a7d1bb4b957c22f738730f383f63f600faddaec686994f            48470 main/binary-amd64/Packages.gz
 8da0d55e45aff81f22f487a0380cea0f03d93d86ea7a9f9512b957730d305a95           199331 main/binary-i386/Packages
 1799348d084364f07f0673f0509d17e699654a54ff7e5b86edaa2e4555e3b90e            39299 main/binary-i386/Packages.gz
 1014404f18e48200f3138d3dd9ed90de9d84244fae13a7596456b5b5449e1f3d           132133 universe/binary-amd64/Packages
 5591e0fb06081cca70cc37735d62171e6f55df098da537c7892fb535accff4ff            31953 universe/binary-amd64/Packages.gz
 c0ff4ae5541267f3c127a9026ba504e2670b0b01945f2d36934adc51e8c56fdf            63809 universe/binary-i386/Packages
 681242e03883fb45dc2439f56da05f8f693c843c612e3996ee96e8c1cc08f823            16063 universe/binary-i386/Packages.gz
SHA512:
 daa84544664d9ba458990305134d185d3d3987a0324e72f2094b446172446377ed375038a766c6c4cc3391d7b34b6b17535f61a3850704a259640d44ab493020           253683 main/binary-amd64/Packages
 d51f1d5e08d745f636e0d9f348b57f6c58cf2eedd1829cddd29df89a75730bc7e7d6f0dc3978a7eb97a304f1df582fab66ca7bb96a3c38bf1d4363a316939cea            48470 main/binary-amd64/Packages.gz
 993f8bf1861b7de0f11f8c658c4afaa3f0ec5997d627e2415b64a90450042f2355e798a15aa0235fceb01adf6c57cd82bfaa50fdd0646b8667d91366dbaf2170           199331 main/binary-i386/Packages
 14ccfa079a89210757d6c404712c705c53063571e81bd1ca9cfbfc8ab43eb7d32a868198ef29ffd300c02aa9d1980e24ed72a5629a1dcf09c7c718d957a141d3            39299 main/binary-i386/Packages.gz
 68ce0176bd004bd8fa5c1ed870f3ffc66c34a22bb9cd8f70e63c0313fe112144ce033d2da694d2608f98940e466966ac19f3bd5c5425f713553c927523bfb7fb           132133 universe/binary-amd64/Packages
 b4e5e031628d8c27f2c50d2a8ac8bb7bb0013e108a3d0b155579cfddcc1ca2c8916c316c5eeb0698c75827a7099af3b745c004227c35bd762bebeacd03b93e83            31953 universe/binary-amd64/Packages.gz
 07dc64111a7d9e8fa0dd7957469cd9f5c30519e3ab4059e523b6c362d14384d820c47e84ff26f2f1c96775bfeaffcf1beb3e9181fd0f381d6ae568738e47beb7            63809 universe/binary-i386/Packages
 07f9669d9eeeaeccd744f16b35be3767f06ce53d09dd7f9132b3b7a0fa0409eff17c6e5c96994c5d6dd4499dd6924ea0e265d6d493a865dd8a107e69ddb6f1e8            16063 universe/binary-i386/Packages.gz

Is this a known issue? How do I fix this?

[gdivis]

Hi @dimas,

I've looked into this and it appears that restricted and multiverse in noble-backports do not actually have any packages in their indexes. It's a quirk of how these connectors are implemented in ProGet that components with no packages indexed are omitted from the output index. Is this causing a problem for you? We can look into changing this behavior, but it likely won't be a trivial fix.

-Greg

[dimas]

@gdivis I confirmed your findings. I have removed the missing components on all my Ubuntu hosts.

My original plan was to have a proxy-like repository that only required changing the repository urls in each host's /etc/sources.list.d configuration files. As a workaround for the connector implementation, I have to monitor the upstream components for packages and update the /etc/sources.list.d configuration on each host in my network as soon as any packages exist.

Just out of curiosity, why does the proget connector omit empty repository components altogether, instead of making them empty like virtually every other Debian repo? It allows one-time configuration on Debian hosts, regardless of whether the component has any packages or not.

[apxltd]

@dimas said in Incomplete proget debian connector local index file for ubuntu noble-backports dist:

why does the proget connector omit empty repository components altogether, instead of making them empty like virtually every other Debian repo?

Most other Debian repos use or fork Dpkg to generate the folder structure based on files on disk.

ProGet doesn't. Instead, we parse/index all packages in the repository in a local sqllite3 database. That database has a "Packages" table, and we use that to generate feed indexes (Release, InRelease). It's possible to change, and involves modifying the sqllite3 databases to add a Components table, and then changing the way we generate all the indexes.

It's not too hard but it's not trivial and probably isn't worth it unless we get more feedback on the matter.

Thanks,
Alex

[dimas]

@apxltd I understand your stance on this. I'm a pretty happy ProGet user. It's better than mirroring the whole repository, so I'm willing to create my own workaround. Thanks for creating such a great product!

[apxltd]

@dimas thanks for that!

We're currently studying/monitoring the performance for public mirroring, and one "concerning" thing I recently saw was a sqllite3 index file explode in file size. We don't know how that happened, but suspect it might have to do with some "aggressive" testing / request cancellation.

Let us know if you spot anything:

https://forums.inedo.com/topic/5410/debian-feed-mirror-performance