Welcome to the Inedo Forums! Check out the Forums Guide for help getting started.
If you are experiencing any issues with the forum software, please visit the Contact Form on our website and let us know!
Debian Feed - Package can only be downloaded by apt once
-
This is an oddly specific issue I'm finding. Background info - fully standard, no customizations
- ProGet is running on Windows Server
- Proget v 2023.29 (Build 17)
- New Debian connector set up with bookworm
- debian feed called
import-deblinked to above connector. - Debian bookworm container setup with private ProGet Debian apt source + key, public Debian feed removed
When I build the container for the first time (works on a VM or any other Debian instance), it will successfully download and install the package from Proget (for discussion,
unixodbc-common. This is good!If I do another build or attempt to install the same package on another computer, apt cannot locate the package. This is bad! I can, however, still see and download the package from the Proget UI and even via curl on the container/vm, ruling out network issues.
If I then delete that package from Proget, and attempt to install via apt -- it works again, but only once. Once the package has been pulled from the Public Debian repo and saved into ProGet, it seems apt cannot find it again.
-
Dockerfile used:
FROM privatedockerrepo/python:3.8-slim RUN apt update RUN apt install -y wget gnupg2 apt-transport-https apt-rdepends RUN echo 'deb https://user:pass@proget.local/debian/import-debian bookworm main contrib non-free' | \ tee /etc/apt/sources.list.d/proget-approved-deb.list RUN wget -qO - https://user:pass@proget.local/debian/import-debian/keys/import-debian.asc | apt-key add - RUN mv /etc/apt/sources.list.d/debian.sources /etc/apt/sources.list.d/debian.sources.old RUN apt update && apt install unixodbc-common CMD ["bash"]Output from first run:
PS> docker build . [+] Building 42.5s (15/15) FINISHED docker:default => [internal] load build definition from Dockerfile => => transferring dockerfile: 1.17kB => [internal] load metadata for privatedockerrepo/python:3.8-slim => [auth] python:pull token for privatedockerrepo => [internal] load .dockerignore => => transferring context: 2B => [internal] load build context => => transferring context: 71B => [1/7] FROM privatedockerrepo/python:3.8-slim => [2/7] RUN apt update => [3/7] RUN apt install -y wget gnupg2 apt-transport-https apt-rdepends => [4/7] RUN echo 'deb https://user:pass@proget.local/debian/import-debian bookworm main contrib non-free' | tee /etc/apt/sources.list.d 0.0s => [5/7] RUN wget -qO - https://user:pass@proget.local/debian/import-debian/keys/import-debian.asc | apt-key add - => [6/7] RUN mv /etc/apt/sources.list.d/debian.sources /etc/apt/sources.list.d/debian.sources.old => [7/7] RUN apt update && apt install unixodbc-common => exporting to image => => exporting layers => => writing image sha256:...Output from second run:
PS> docker build . [+] Building 42.5s (15/15) FINISHED docker:default => [internal] load build definition from Dockerfile => => transferring dockerfile: 1.17kB => [internal] load metadata for privatedockerrepo/python:3.8-slim => [auth] python:pull token for privatedockerrepo => [internal] load .dockerignore => => transferring context: 2B => [internal] load build context => => transferring context: 71B => [1/7] FROM privatedockerrepo/python:3.8-slim => [2/7] RUN apt update => [3/7] RUN apt install -y wget gnupg2 apt-transport-https apt-rdepends => [4/7] RUN echo 'deb https://user:pass@proget.local/debian/import-debian bookworm main contrib non-free' | tee /etc/apt/sources.list.d 0.0s => [5/7] RUN wget -qO - https://user:pass@proget.local/debian/import-debian/keys/import-debian.asc | apt-key add - => [6/7] RUN mv /etc/apt/sources.list.d/debian.sources /etc/apt/sources.list.d/debian.sources.old => ERROR [7/7] RUN apt update && apt install unixodbc-common ------ > [7/7] RUN apt update && apt install unixodbc-common: 0.413 0.413 WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 0.413 29.70 Get:1 https://proget.local/debian/import-debian bookworm InRelease [39.1 kB] 30.13 Get:2 https://proget.local/debian/import-debian bookworm/non-free all Packages [53.7 kB] 30.50 Get:3 https://proget.local/debian/import-debian bookworm/contrib all Packages [27.7 kB] 30.74 Get:4 https://proget.local/debian/import-debian bookworm/contrib amd64 Packages [66.1 kB] 31.02 Get:5 https://proget.local/debian/import-debian bookworm/non-free amd64 Packages [125 kB] 31.27 Get:6 https://proget.local/debian/import-debian bookworm/main amd64 Packages [10.6 MB] 36.09 Get:7 https://proget.local/debian/import-debian bookworm/main all Packages [5719 kB] 38.67 Fetched 16.7 MB in 38s (436 kB/s) 38.67 Reading package lists... 38.99 Building dependency tree... 39.08 Reading state information... 39.09 21 packages can be upgraded. Run 'apt list --upgradable' to see them. 39.09 W: https://user:pass@proget.local/debian/import-debian/dists/bookworm/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details. 39.10 39.10 WARNING: apt does not have a stable CLI interface. Use with caution in scripts. 39.10 39.10 Reading package lists... 39.42 Building dependency tree... 39.51 Reading state information... 39.52 E: Unable to locate package unixodbc-common ------ Dockerfile:16 -------------------- 14 | RUN wget -qO - https://user:pass@proget.local/debian/import-debian/keys/import-debian.asc | apt-key add - 15 | RUN mv /etc/apt/sources.list.d/debian.sources /etc/apt/sources.list.d/debian.sources.old 16 | >>> RUN apt update && apt install unixodbc-common 17 | 18 | CMD ["bash"] -------------------- ERROR: failed to solve: process "/bin/sh -c apt update && apt install unixodbc-common" did not complete successfully: exit code: 100 =>
-
Hi @dan-brown_0128,
That is odd! We'll see if we can reproduce it and let you know what we find.
-Greg
-
Hi @dan-brown_0128,
I've tried to reproduce this both inside of Docker and on a clean bookworm vm, and haven't had any luck. It works for me whether the package has been pulled to ProGet or not, no matter how many times I try. Unfortunately I don't have any ideas what might be happening here.
-Greg
-
Hi @dan-brown_0128 , @gdivis ,
We received a support ticket that seems to be a similar topic...
I am using a ProGet Debian feed to mirror http://deb.debian.org/debian. I am using it on Debian containers running on my desktop. I have noticed that packages that I had installed already once are no longer available for a second installation. I have attached a screenshot that uses the command "apt update && apt install -y git" as example. According to the error message, the package "git" is unknown.
However when looking it up in the ProGet feed, it is listed as a cached package as it was installed in an earlier execution of the container. I have then set a retention rule on the feed to delete absolutely all packages and run the job manually to clean all packages cached on the feed. When re-executing the exact same command, the package git and its dependencies are found and installed.It appears that cached packages are no longer returned to the Linux instance when executing "apt update". I can force a complete cleanup of the ProGet feed prior to an installation as a workaround but this is a bit tedious.
So I wonder if this has something to do with how we're generating indexes, or something to that effect?
I also asked the other user for input - but we're currently stuck trying to reproduce this, so any other insight would be helpful.
Steve
-
Hi @dan-brown_0128 , @stevedennis
I had that thought too, and I made sure caching was enabled when I tried to reproduce this. I have some more ideas for trying to recreate this today - hopefully with more luck this time.
-
I'm starting to think this is related to package versions with a
+. We can keep going down the road ofunixodbc-common_2.3.11-2+deb12u1_all.debfor example.If I disable the cache and do not pull packages to proget, I can download/install just fine.
If I promote the package to another feed, I cannot install it. BUT if I try a package without a
+in the version/filename in that second feed, it does install or at least resolve (ex: bcal)There is one big difference in the index that might be the key... The original/upstream Debian index does not urlencode the
filenamebut it looks like ProGet isdebian upstream:
Filename: pool/main/u/unixodbc/unixodbc-common_2.3.11-2+deb12u1_all.deb
Proget:Filename: pool/bookworm/main/all/unixodbc-common/unixodbc-common_2.3.11-2%2Bdeb12u1_all.deb
-
Oh and also to note: there is no network traffic (captured by our networking team) or from IIS logs when attempting to
apt installunixodbc-common from ProGet (pulled to feed). Maybe apt is thinking that filename is invalid and skips it
-
Good find!
I did my testing against
unixodbc-common_2.3.11-2+deb12u1_all.deb, and oddly everything worked- I wonder if this behavior varies in different versions of apt?Anyway, we should change ProGet to match the Debian index behavior, so I will file this as a bug.
-
Hi @dan-brown_0128,
I've done some more research on this, and per the spec we definitely should not be url encoding anything in the Filename field: https://wiki.debian.org/DebianRepository/Format#Filename
I've filed this as PG-2591. It's an easy change, so we'll include it in this week's release on Friday. It still is strange that I couldn't reproduce this, but hopefully this will resolve the issue for you and anyone else that is seeing this.
-
Quick update -- Just installed 2023.30 in our test environment. At first I was still not able to download packages with
+in the version (despite the index being correct). After clearing the feed and re-promoting packages it does seem to work now!
-
Great to hear! Thanks for the followup
