apt Package file malformatted

frei_zs

Hey everyone,

we are using Version 2023.32 (Build 3) of ProGet with the new Debian feeds and I came across an issue with apt-cache not being able to find all packages I uploaded to the feed.

There are a total of 31 packages in the feed, but when looking for them with apt-cache search <string> only two showed up in the output.

So I investigated where and why and after looking into the
proget.<hostname>_debian_debian-test_dists_bookworm_main_binary-amd64_Packages
file and compared that with the one I for example got from my aptly repo, I saw that there are no spaces / newlines between packages. I manually edited the file, created a newline between each package and now apt-cache finds the packages.

Is this some bug? All other Packages-files in the /var/lib/apt/lists folder do have that convention to seperate package information with an additional newline.

Many thanks,
Frank

atripp

Hi @frei_zs ,

Unfortunately we don't know what aptly is doing behind-the-scenes and can't guess what could be going on here. I don't believe that ProGet is generating that file; it looks like something aptly is doing based on data downloaded from ProGet.

ProGet implements a Debian repository as specified here:
https://wiki.debian.org/DebianRepository

It's possible that there's a bug in our implementation, or a bug in aptly, or both. But we can't really track it down with the information here :(

Your best bet would be to start with the aptly team, and to see if they have any idea.

Cheers,
Alana

frei_zs

Hey Alana,

thank you, but I would take aptly out of the bill here, as I just use it to build the deb-packages.
Those are then uploaded to the ProGet-Debian-Feed per curl.

The Packages-files is provided by ProGet not aptly? As soon as the deb-Packages are build, aptly's job is done. The rest is handled by ProGet. It builds the InRelease and Packages file, which apt then downloads per apt update from the ProGet-Endpoint, that is configured on the client system per "Integrate with apt" instructions:

wget -O "testfeed.gpg" https://proget.<hostname>/debian/tomedo-test/keys/testfeed.asc && sudo apt-key add "testfeed.gpg"

echo "deb https://proget.<hostname>/debian/tomedo-test/ bookworm main" | sudo tee "/etc/apt/sources.list.d/proget-tomedo-test.list"

I don't think aptly interferes with anything here? My aptly is a different machine hosting the deb-packages source and only used for that purpose. There is besides the curl upload to the feed no connection between proget and aptly.

Many thanks,
Frank

atripp

Hi @frei_zs ,

Thanks for clarifying; I'm not at all familiar with the ecosystem and I really don't know how to use deb/aptly/etc. I also don't know what a Packages-files file is.

However, I can say that ProGet provides provides endpoints for two "files":

These are dynamically generated by ProGet, and we've not run into any issues nor have other users expressed issues. That doesn't mean they're error-free, it just means we don't know what's wrong since we tried to follow the spec and it worked for us/others.

I don't know what the URL is for these offhand, but if you use a traffic capturing tool, you can see exactly what deb is requesting.

If you can help us identify what's wrong with the output generated by ProGet, there's probably good chance we can fix them. We'd also want to know why it "works on our machine" and others.

Thanks,
Alana

frei_zs

Hey Alana,

that is a good question, why neither you nor other users ran into this issue.
I'm wondering what might be different with my setup. So I was trying with a fresh / unaltered
Debian 12 VM with the netinst.iso, but have the same issue there.

The Packages-file I refer to is the ""Packages" Indices" from your referenced documentation:
https://wiki.debian.org/DebianRepository/Format#A.22Packages.22_Indices

I've read through this and more of Debians docs, but there is no explicit statement that entries in that file have to be seperated by an additional newline. All I could find is
"Each stanza shall begin with a "Package" field. Clients may also accept files where this is not the case."

It is only a small thing, which I fix by running this command to alter the Packages indices file from ProGet:

sed -i 's/Package:/\nPackage:/g' /var/lib/apt/lists/proget.hostname.domain_debian_tomedo-test_dists_bookworm_main_binary-amd64_Packages

which adds a newline between each Package entry.

If we look at the Packages indices provided by Debian itself, it looks like this for example:

root@myhost# less /var/lib/apt/lists/ftp.de.debian.org_debian_dists_bookworm_main_binary-amd64_Packages

Package: 0ad
Version: 0.0.26-3
Installed-Size: 28591
Maintainer: Debian Games Team <pkg-games-devel@lists.alioth.debian.org>
Architecture: amd64
Depends: 0ad-data (>= 0.0.26), 0ad-data (<= 0.0.26-3), 0ad-data-common (>= 0.0.26), 0ad-data-common (<= 0.0.26-3), libboost-filesystem1.74.0 (>= 1.74.0), libc6 (>= 2.34), libcurl3-gnutls (>= 7.32.0), libenet7, libfmt9 (>= 9.1.0+ds1), libfreetype6 (>= 2.2.1), libgcc-s1 (>= 3.4), libgloox18 (>= 1.0.24), libicu72 (>= 72.1~rc-1~), libminiupnpc17 (>= 1.9.20140610), libopenal1 (>= 1.14), libpng16-16 (>= 1.6.2-1), libsdl2-2.0-0 (>= 2.0.12), libsodium23 (>= 1.0.14), libstdc++6 (>= 12), libvorbisfile3 (>= 1.1.2), libwxbase3.2-1 (>= 3.2.1+dfsg), libwxgtk-gl3.2-1 (>= 3.2.1+dfsg), libwxgtk3.2-1 (>= 3.2.1+dfsg-2), libx11-6, libxml2 (>= 2.9.0), zlib1g (>= 1:1.2.0)
Pre-Depends: dpkg (>= 1.15.6~)
Description: Real-time strategy game of ancient warfare
Homepage: https://play0ad.com/
Description-md5: d943033bedada21853d2ae54a2578a7b
Tag: game::strategy, interface::graphical, interface::x11, role::program,
 uitoolkit::sdl, uitoolkit::wxwidgets, use::gameplaying,
 x11::application
Section: games
Priority: optional
Filename: pool/main/0/0ad/0ad_0.0.26-3_amd64.deb
Size: 7891488
MD5sum: 4d471183a39a3a11d00cd35bf9f6803d
SHA256: 3a2118df47bf3f04285649f0455c2fc6fe2dc7f0b237073038aa00af41f0d5f2

Package: 0ad-data
Version: 0.0.26-1
Installed-Size: 3218736
Maintainer: Debian Games Team <pkg-games-devel@lists.alioth.debian.org>
Architecture: all
Pre-Depends: dpkg (>= 1.15.6~)
Suggests: 0ad
Description: Real-time strategy game of ancient warfare (data files)
Homepage: https://play0ad.com/
Description-md5: 26581e685027d5ae84824362a4ba59ee
Tag: role::app-data
Section: games
Priority: optional
Filename: pool/main/0/0ad-data/0ad-data_0.0.26-1_all.deb
Size: 1377557908
MD5sum: fc5ed8a20ce1861950c7ed3a5a615be0
SHA256: 53745ae74d05bccf6783400fa98f3932b21729ab9d2e86151aa2c331c3455178

For the file provided by ProGet the output is this per default:
less /var/lib/apt/lists/proget.hostname.domain_debian_tomedo-test_dists_bookworm_main_binary-amd64_Packages

Package: tomedo-kim
Version: 0.2-1
Section: tomedo
Priority: optional
Architecture: amd64
Maintainer: Zollsoft GmbH <info@zollsoft.de>
Description: Installs KIM systemd service
Filename: pool/bookworm/main/amd64/tomedo-kim/tomedo-kim_0.2-1_amd64.deb
Size: 219494592
MD5sum: 6ef854e2dce4eb1e6dc29e6d6da126b2
SHA1: bffec83a97bd415dad18b9b71a0d5f96ed9633e7
SHA256: 542bede9a9c243da34d2977ede4ee3c8f3aada81343f18d5d405fcc4acf4c8fb
SHA512: 91334edf6770f218aadcc73ea4a0c76fd4fe4ed365b758a572fcf386ffc82eb063b42e7b98837181cfec05142b4c16a042f7f9455190541227f65dff99f25d76
Package: tomedo-nginx
Version: 1.1-1
Section: tomedo
Priority: optional
Architecture: amd64
Pre-Depends: nginx-full, tomedo-certificates, netcat-openbsd
Maintainer: Zollsoft GmbH <info@zollsoft.de>
Description: Installs nginx web server which acts as a proxy for other web services
Filename: pool/bookworm/main/amd64/tomedo-nginx/tomedo-nginx_1.1-1_amd64.deb
Size: 20672
MD5sum: 3c39f4ed47edb8e94509ba4264bcd789
SHA1: 3e25a5d5be57115941ecb53fd26a7dcfe21fd5ce
SHA256: 351b02c1b4e16da5ad4a2f5f476b90d63954551929eb9c1b8e42b5a494746bc6
SHA512: 5d2e356ce7fe5dbe8adb1cf10f3d67bd316ce6cae57f11e87275fe5996aff0a7be3541d1598aaf736ce7e74042a6a4b3d266c161886e548ea6739f289ec856c0

There is no newline / empty line between the two package definitions.
Changing this with the above sed command I alter it to:

Package: tomedo-kim
Version: 0.2-1
Section: tomedo
Priority: optional
Architecture: amd64
Maintainer: Zollsoft GmbH <info@zollsoft.de>
Description: Installs KIM systemd service
Filename: pool/bookworm/main/amd64/tomedo-kim/tomedo-kim_0.2-1_amd64.deb
Size: 219494592
MD5sum: 6ef854e2dce4eb1e6dc29e6d6da126b2
SHA1: bffec83a97bd415dad18b9b71a0d5f96ed9633e7
SHA256: 542bede9a9c243da34d2977ede4ee3c8f3aada81343f18d5d405fcc4acf4c8fb
SHA512: 91334edf6770f218aadcc73ea4a0c76fd4fe4ed365b758a572fcf386ffc82eb063b42e7b98837181cfec05142b4c16a042f7f9455190541227f65dff99f25d76

Package: tomedo-nginx
Version: 1.1-1
Section: tomedo
Priority: optional
Architecture: amd64
Pre-Depends: nginx-full, tomedo-certificates, netcat-openbsd
Maintainer: Zollsoft GmbH <info@zollsoft.de>
Description: Installs nginx web server which acts as a proxy for other web services
Filename: pool/bookworm/main/amd64/tomedo-nginx/tomedo-nginx_1.1-1_amd64.deb
Size: 20672
MD5sum: 3c39f4ed47edb8e94509ba4264bcd789
SHA1: 3e25a5d5be57115941ecb53fd26a7dcfe21fd5ce
SHA256: 351b02c1b4e16da5ad4a2f5f476b90d63954551929eb9c1b8e42b5a494746bc6
SHA512: 5d2e356ce7fe5dbe8adb1cf10f3d67bd316ce6cae57f11e87275fe5996aff0a7be3541d1598aaf736ce7e74042a6a4b3d266c161886e548ea6739f289ec856c0

which then works fine.

For all files provided by Debian there is this additional empty line between packages.
This is another one for example:
less /var/lib/apt/lists/ftp.debian.org_debian_dists_bookworm-backports_contrib_binary-amd64_Packages

Package: diaspora-installer
Version: 0.7.18.2+debian3~bpo12+1
Installed-Size: 111
Maintainer: Debian Ruby Extras Maintainers <pkg-ruby-extras-maintainers@lists.alioth.debian.org>
Architecture: all
Replaces: diaspora
Depends: build-essential, diaspora-common (= 0.7.18.2+debian3~bpo12+1), ghostscript, imagemagick, libcurl4-openssl-dev, libffi-dev, libmagickwand-dev, libpq-dev, libruby3.1, libssl-dev, libxml2-dev, libxslt-dev, rsync, ruby-bundler, ruby-dev, ruby-http-parser, tzdata, wget, zlib1g-dev
Conflicts: diaspora, libruby2.7
Description: distributed social networking service - installer
Homepage: https://wiki.debian.org/Diaspora
Description-md5: 120c7fb469648e6eb3e942926e8b1ea0
Section: contrib/net
Priority: optional
Filename: pool/contrib/d/diaspora-installer/diaspora-installer_0.7.18.2+debian3~bpo12+1_all.deb
Size: 21584
SHA256: f71649c6221b647dca57e7c8e9c045a4544a08103e48714d28465d545a1139c9

Package: diaspora-installer-mysql
Source: diaspora-installer
Version: 0.7.18.2+debian3~bpo12+1
Installed-Size: 12
Maintainer: Debian Ruby Extras Maintainers <pkg-ruby-extras-maintainers@lists.alioth.debian.org>
Architecture: all
Depends: dbconfig-mysql | dbconfig-no-thanks, default-libmysqlclient-dev, default-mysql-server | virtual-mysql-server, diaspora-installer (= 0.7.18.2+debian3~bpo12+1)
Description: distributed social networking service - installer (with MySQL)
Homepage: https://wiki.debian.org/Diaspora
Description-md5: c18a155ca94ebb0900ab3ea2aa77e4b9
Section: contrib/ruby
Priority: optional
Filename: pool/contrib/d/diaspora-installer/diaspora-installer-mysql_0.7.18.2+debian3~bpo12+1_all.deb
Size: 4852
SHA256: 05e50f43d84a1cbff57c4019d867d6677007d1307f9202f6005a117192f40a70

I will further investigate to find out why this seems to work for you and others but not for me, but I think having that additional seperation between packages is not a bad thing, as it follows the (likely undocumented) format for Packages indices more strictly and prevents issues (maybe only for me 8).

Many thanks,
Frank

atripp

Thank you very much for the detailed analysis @frei_zs !

I agree, this seems like it's worth trying. And hopefully it won't break :)

We'll try to get this in the next maintenance release on Friday via PG-2618; if it's something we think is worth a pre-release/test cycle, we'll let you know, but probably simple and harmless enough to do.

Clearly "their" files have new lines, and the specs don't say it's "wrong" so far as I can tell.

frei_zs

Thank you for taking time to sort this out with me!

I greatly appreciate your efforts and will be happy, if this works out well. :)