Uploading ca-certificates (2023 version) to a ProGet conda feed does not work

e.rotteveel_1850

I wanted to upload the ca-certificates (2023.01.10) package to a conda feed. The upload does not give any errors. On the 'packages' page (right from ProGet main page), the package is shown. But clicking on it gives a 404 error.

I also tried the 2022.10.11 version, and that one did work. Next, the 2022.07.19 does not work.... But the 2022.3.29 version just works.

It seems ProGet does not support packages with a calendar versioning scheme with leading zeros for the month/minor number.

I checked the storage, and ProGet is actually renaming the packages! In the storage, I have ca-certificates-2023.1.10-haa95532_0.tar.bz2, while the original filename is ca-certificates-2023.01.10-haa95532_0.tar.bz2 (including the zero). Meanwhile, ProGet does store the leading zero in its database, looking at the "packages" page.

Summarizing: there is a mismatch between what ProGet is putting in it's database and what ProGet is putting in it's storage for conda package with calendar version scheme. In my opinion, ProGet should not/never change the version of packages, even if it is to remove leading zeros. I hope you agree

In any case, thanks for looking into this issue! I hope it can get fixed.

EDIT: We're currently (still) on ProGet 6.0.20, realised that there have been quite some new versions already. Will check the release notes first.

e.rotteveel_1850

Came across this issue on YouTrack of which the state is "completed":
https://inedo.myjetbrains.com/youtrack/issue/PG-2220/FIX-Error-with-Conda-package-versions-that-have-leading-0s

However, it still does not work. I tested with Proget 2022.0.19, but the file is still renamed to "2023.1.10" while the ProGet packages page shows "2023.01.10".

I think that issue should be re-opened.

atripp

Hi @e-rotteveel_1850 ,

The underlying problem is that ca-certificates has invalid package versions, at least according to Conda's own versioning specification.

2022.07.19 is not supposed to be permitted in a repository... and yet it's there. ProGet follows the Conda specification, which says packages with leading zeros should be "normalized" to 2022.7.19. I guess they treat their specs more like "guidance" than "specifications"

In any case, as you noted, this was addressed in a newer version (PG-2220 in ProGet 2022.10), so upgrading should take care of it.

Cheers,
Alana

atripp

Whoops, posted too fast :)

@e-rotteveel_1850 said in Uploading ca-certificates (2023 version) to a ProGet conda feed does not work:

I tested with Proget 2022.0.19, but the file is still renamed to "2023.1.10"

The package store is internal to ProGet, and we don't support accessing or modifying those files directly. The folder structure or naming of the files won't impact usage in the ProGet UI or API.

e.rotteveel_1850

@atripp Thanks for your quick replies.

Are you sure 2023.01.10 should work? Because it doesn't work with ProGet 2022.0.19. I installed the latest version of ProGet, created a new conda feed, uploaded ca-certificates 2023.01.10. However, it is not uploaded/posted correctly.

In the GUI, the feed overview says that there are no local packages in the feed.
Going from the main page to packages, I can see the newly uploaded package (version indicates 2023.01.10 - which it also did with ProGet 6.0.20). Clicking on the package link gives me a 404 error.

Also tested with ca-certificates 2022.07.19, same issue.

If I use conda search -c CHANNEL_URL ca-certificates I see the package popping up, with version 2023.1.10, so without the leading zero. The package URL also contains the version number without leading zero. But trying to download that URL directly, does not work. However - and this is interesting I think - if I change the URL to include the leading zero, the download works!

But after starting the download, ProGet tries to forward me to the URL without the leading zero, giving me "package does not exist". Yet, I was able to download the files (although the downloaded file misses the leading zero, i.e. it changed the version)

I would not care too much about the GUI normally. Unfortunately, conda wants to use the URL from the index which is wrong so conda cannot download the package.

Overall, it seems the issue was not entirely fixed. I hope you can look at it again, thanks!

ps. looking at PEP440 - I don't see it explicitly excluding/forbidding the use of leading zeros. Actually, it even uses a version 2014.04 as an example. Then looking at conda's package specs: https://docs.conda.io/projects/conda/en/latest/user-guide/concepts/pkg-specs.html, again there is the example 1996.07 and I don't see it excluding leading zeros either.... Am I missing something?

atripp

Hi @e-rotteveel_1850 ,

I haven't really tested this, just reporting on information I found in our notes :)

The problem that we fixed via PG-2220 was reproduceable as follows:

Create Conda feed w/ connector to default public repository
Access this page in UI: feeds/MyCondaFeed/ca-certificates/versions

It also gave errors in the ProGet API. But the underlying issue was related to unexpected ("invalid") metadata from the remote Conda repository's API (index files), specifically with sorting (comparing) those leading zeros.

Python specifications give me a headache, but it has something to do with the PEP-440 Normalization Rules not being followed under the hood

Regardless, it sounds like this is a different bug...

Can you identify how we can repro/fix (without using CONDA client)?

Create new CONDA feed (no connector)
Download ???? files from the public site (I couldn't tell from this page)
Upload those files to ProGet
Web UI shows ???? instead of ???
API url ???? doesn't download file (as expected)

Cheers,
Alana

e.rotteveel_1850

@atripp Thanks again, I'm happy to help a bit here.

I think it is a different bug indeed, may be some inconsistencies in ProGet's inner workings. The steps to reproduce:

Create new CONDA feed without connector (I was using private/internal). Let's name it conda_channel
Download this file: https://anaconda.org/anaconda/ca-certificates/2023.01.10/download/win-64/ca-certificates-2023.01.10-haa95532_0.tar.bz2
Upload to the feed we just created
Now note: WebUI of the feed says there are no packages (error 1)
Check this file: PROGET_BASE_URL/conda/conda_channel/win-64/repodata.json
Note that the keys under the packages structure have their leading zeros removed.
Try to download: PROGET_BASE_URL/conda/conda_channel/win-64/ca-certificates-2023.1.10-haa95532_0.tar.bz2 => error
Add leading zero to URL => PROGET_BASE_URL/conda/conda_channel/win-64/ca-certificates-2023.01.10-haa95532_0.tar.bz2 => works

The issue is that conda combines the URL up to win-64 with the package name. And although ProGet saves the file without the leading zero, the URL is not correctly resolved for some reason.

GUI issues:

This page: PROGET_BASE_URL/feeds/conda_channel/ca-certificates/versions => gives 500 error
On feed page: PROGET_BASE_URL/feeds/conda_channel => you cannot see the package

Conda issue:

Fortunately, the conda client does not care too much about literal versioning and removal of leading zeros. If I say I want version "2023.01.10", but on the ProGet server it has "2023.1.10", it will properly resolve to the correct package entry.
But if the endpoint/URL is not working, conda cannot download the package.
The problem is that the endpoint that ProGet offers is inconsistent in some way with its storage/database. See steps 7 and 8:
-- although it removes the leading zeros before storing, the database-entry still includes (it seems) the leading zero.
-- The download endpoint only responds to the URL including the leading zero, but downloads the file without leading zero
-- The file storage does not have the leading zero

To get conda client to work with minimal effort, I think the only fix is to make the endpoint work properly (i.e. endpoint without the leading zero should work, as used in step 7).

For the GUI, I don't know the details of course. However, it seems that some things/actions just store the raw/original version, while other parts are storing a normalized version.

Regarding PEP440 normalization: I think conda normally just stores stuff as-is and applies normalization "on-thefly" when sorting/comparing is needed, for instance.

Btw - just noticed another case where the same issue occurs. A package with version "4.3.0a" is changed to "4.3.0a0". Which is normalization according to PEP440 - although just "4.3.0a" is not strictly forbidden. But again, the URL to download the package (this time including the added zero) does not work, but if I change the URL (change to literal "4.3.0a") it will work.

atripp

Thanks @e-rotteveel_1850 , these repro steps will be very helpful to debug/fix the problem.

With platforms we know little about (Python, Conda) figuring out the repro steps is often the hardest part

Please stay tuned; we'll post an update once we identify or fix the problem

e.rotteveel_1850

Great, thanks! Glad to be of help.

Meanwhile, another kind of version detail that gives issues:
https://peps.python.org/pep-0440/#pre-release-separators

I gave one of my own package the version 2023.2.beta4. According to the above, it should be acceptable to have pre-release separators and also write beta instead of just b. ProGet adjusts it to 2023.2b4 which is the correct normalization. Again, similar to the above, the download URL works if I change it back to 2023.2.beta4 although ProGet stores it as 2023.2b4 internally.

Actually for this one, the conda client is wrong as well: searching for 2023.2.beta4 does not work. The conda client itself does not correctly normalize pre-release versions, it seems. But that's not for you to worry about :)

Looking forward to the fix, I'll watch this thread and test it once available.

atripp

Thanks again for these reproduction instructions! I confirmed the behavior very easily, though it doesn't look like a trivial fix (at least to me)

We are targeting PG-2278 for the upcoming maintenance release (Friday), but it might be delayed if we have other priorities until the following maintenance (Feb 24). Of course we can provide a prerelease/patch version as soon as we code the fix :)

gdivis

It turns out the problem was in the code that adds the package record to the database was using the raw version string instead of the normalized version, which causes a lot of problems as you can see. We've fixed that in PG-2278, and this fix will be in ProGet 2022.20 to be released this Friday. Unfortunately it won't fix any invalid records that have already been created. Time permitting, we will try to get a job in that can be run to fix nonnormallized versions in the database, but if it doesn't make it in this release you'll need to manually remove or update that row in the SQL database. Let us know if you need any assistance with this.

e.rotteveel_1850

@atripp @gdivis Thank you very much, I will test the new version early next week then and let you know if all is working as expected.

As for repairing existing entries: That should be OK I think, its only a small number of packages that have problems. And with a print of the repository and some regex, It should be easy to identify those.

e.rotteveel_1850

I tested the new version (local, brand-new installation) and now all corrections to version numbers are consistent. Thanks for the quick fix!