Posts made by e.rotteveel_1850

Sorry for my late reply, but we have since installed version 2023.16 and repodata.json works perfectly fine now! Thanks for including this in your updates.

Conda uses the "If-Modified-Since" header to know if the repodata should be downloaded (or not), but it seems the ProGet server does not provide the correct value for the "Last-Modified" response header.

I just uploaded a new package to a channel (both via API and a different one via manual upload), but the Last-Modified header says repodata.json (and channeldata.json) were last modified on February 23....

Thanks for looking into this!
We're currently on ProGet 2023.16

Thanks for the effort again!

I just tried with one of the numpy packages on the main conda channel:
https://repo.anaconda.com/pkgs/main/win-64/numpy-1.23.5-py39h6917f2d_1.tar.bz2

And I get the same problem with the win-64/repodata.json URL.

It went okay with one of our own packages though (but not with another). But even after comparing the metadata for both packages in detail I was not able to find problematic data or characters (none that I could think of).

Looking forward to the fix!

Conda has the concept of package builds on top of package versions. I can image other package managers offer similar features. The idea is that you can build your package in different ways, linking it to different versions of binary libraries, for instance. You keep the same version because your source code is the same, but the build-string (conda generates a hash from your dependencies) differs, and you produce multiple packages.

This seems to work OK-ish now in ProGet. Using the manual upload in the WebGUI, I can upload a package with a version that is already on the server, but with a different build-number/string. But if I try this via CURL (and an API key), it does not work.

I uploaded some example files here:
https://nextcloud.marin.nl/index.php/s/7zBHSDx8QYLCzfJ

If you create an empty feed, then first upload proget-package-2023.8.0-1.tar.bz2, that will work fine using either CURL or manual upload in the WebGUI. But if you then try to upload proget-package-2023.8.0-3.tar.bz2, it will work in the WebGUI but not via CURL. Via CURL, I get the error:

The package win-64/proget-package.2023.8.0 already exists and the current user or service does not have the Feeds_OverwritePackage privilege.

I am using a feed API Key with CURL (so --user api:API_KEY) for which all permissions have been enabled. But there should be no message of 'overwriting' a package at all, because it is a different build.

Thanks again for looking into this. If you need more information, I'm happy to help.

First of all, @atripp : the constrains (i.e. optional dependencies) appears to be working. Thank you and your team! We're now trying to setup conda feeds with the 2023 version of ProGet and we can see the conda feed features improved!

However, the conda feed breaks down after uploading some packages. I cannot really find out what triggers the problem exactly, but I uploaded some example packages here:

https://nextcloud.marin.nl/index.php/s/7zBHSDx8QYLCzfJ

When I create a new, empty conda feed, the repodata.json works fine. I can go to one of:

• https://my.proget.url/conda/my-conda-feed/win-64/repodata.json
• https://my.proget.url/conda/my-conda-feed/noarch/repodata.json

And they look fine. Of course there are no packages yet.

If I upload the hello-inedo-triple package from my examples (via the manual upload in the WebGUI), the second link (noarch repodata.json) still works. But if I upload proget-package-2023.8.0-1.tar.bz2, the win-64 repodata.json link responds with:

Object reference not set to an instance of an object.

Meanwhile, the channeldata.json (https://my.proget.url/conda/my-conda-feed/channeldata.json) works perfectly fine.

I hope you can look into this, thanks!

Thanks, looking forward to the next release!

With the 'constrains' option working, we can really use ProGet as our main method of distributing (python)packages within our company. It currently works great for most packages, except for so-called meta-packages with optional dependencies.

Meta-packages are nothing more than a list of dependencies. The great thing about them is that they can very effectively "lock" environments meaning that users cannot (either by accident or on purpose) install different versions of packages critical to our workflows. The anaconda package is also a meta-package. If you want to install some package with a different version than stated in the meta-package, you will need to uninstall that meta-package first.

But there are plenty of cases where you don't want users/developers to have to install all 400+ packages of interest each time they create a new conda environment. For that, the "constrains" option comes in handy: we still strongly fixate the versions of certain packages, but they do not have to be installed directly. Only when a user needs such a package later on, he will get the version we want him to have. This is great when someone starts a new project: initially they only need pandas and/or numpy, but later they want to try some machine learning and need scikit-learn or pytorch.

Good luck with the development of version 2023, I'll keep an eye on new releases.

I now changed to 2022.25, and CURL uploads are working again. Thanks!

Thanks for the quick reply and action!

Thanks for your reply! For these features, we'll wait for the new version. I saw on a blog page that you expect a release in Q2 (maybe Q3). Is that still the planning?

Hello again, since the new version (which solved some of the conda-related issues), we are now facing problems with CURL uploads. When uploading a *.tar.bz2 file to a conda feed, it gives the error: The content has not been fully buffered yet.

Looking in the ProGet logs via the UI, I see these messages:

System.NotSupportedException: The content has not been fully buffered yet.
at Microsoft.AspNetCore.WebUtilities.FileBufferingReadStream.Seek(Int64 offset, SeekOrigin origin)
at Inedo.IO.UndisposableStream.Seek(Int64 offset, SeekOrigin origin)
at System.IO.Compression.ZipArchive.ReadEndOfCentralDirectory()
at System.IO.Compression.ZipArchive..ctor(Stream stream, ZipArchiveMode mode, Boolean leaveOpen, Encoding entryNameEncoding)
at System.IO.Compression.ZipArchive..ctor(Stream stream, ZipArchiveMode mode, Boolean leaveOpen)
at Inedo.ProGet.Feeds.Conda.CondaPackageMetadata.TryOpenZip(Stream stream)
at Inedo.ProGet.Feeds.Conda.CondaPackageMetadata.ReadFromPackage(Stream stream, Stream& icon)
at Inedo.ProGet.Feeds.Conda.CondaFeed.InstallPackageAsync(Stream stream, InstallPackageOptions options, Boolean promotion, Nullable`1 downloadCount, Nullable`1 publishDate, CancellationToken cancellationToken)
at Inedo.ProGet.WebApplication.FeedEndpoints.Conda.CondaFeedHandler.ProcessRequestAsync(HttpContext context, WebApiContext apiContext, CondaFeed feed, String relativeUrl)
at Inedo.ProGet.WebApplication.FeedEndpoints.FeedEndpointHandler.FeedRequestHandler.ProcessRequestAsync(HttpContext context)

It appears to happen with any conda package I try to upload via CURL. Meanwhile, manual upload via the web-UI just work fine.

For debugging purposes, here is a package that leads to the errors:
https://nextcloud.marin.nl/index.php/s/LSXc4AQYkBmC3zQ

I have justed tested some uploads with 2022.24 and it seems to work fine now, thanks!

When I use the ProGet UI to go to the page of a specific version of a package, I can delete either that specific version or "all package versions".

However, with conda (and I guess with any package manager) you can have different builds of the same package version. This is not limited to different builds for different OS'es, but also different builds that each have slightly different dependencies**.

Does ProGet "understand" the idea of different builds for the same version of a package?

In ProGet UI, I can see a list of files per package, per version, but I don't see a way to just delete one of those files. Is this maybe possible via the API?

** about the example of dependencies: Let's say I have a package that needs the HDF5 library, but I want my package to be backwards compatible (even for latest updates). In that case, I could build my package for version 1.X and link it to different versions of the HDF5 lib, thus creating multiple builds with the same version, but a package manager on a users PC will select the proper build depending on the available HDF5 library.

Thanks for the quick reply! I'll wait for the new release and give it a try then. No hurry for this, I just noted it happening when uploading some package to it. For now, I can just prevent adding such characters to my package description/summary.

I went through Conda's code to see how it handles the channeldata.json file. Basically, you can use this to reproduce the problem:

import requests
import json

r = requests.get('http://PROGET.BASE.URL/conda/conda_channel/channeldata.json')

with open('some_file', 'wb') as f:
    f.write(r.content)

with open('some_file') as f2:
    A = f2.read()

Furthermore, looking at Anaconda's main repository and the channeldata.json in there:
https://repo.anaconda.com/pkgs/main/channeldata.json
You will also find quite some "encoded" characters (search for "u201c" for instance)

I think conda/conda-build normally make sure the channeldata.json file is as plain as possible. That should at least apply to the description and summary keys, but maybe to the whole file in general. It probably also applies to the repodata.json file in the sub-directories such as noarch, win-64 and linux-64.

In the channeldata.json file (which can be found at http://PROGET.BASE.URL/conda/channel/channeldata.json), packages with non ANSI or non ASCII characters lead to parsing problems in conda-build.

I created a package that allows to reproduce the problem, and show the differences with Conda channels produced via conda build. The example package can be downloaded here: https://nextcloud.marin.nl/index.php/s/QCkmijoD5oiJYx9

If I created (using the latest ProGet version) a new conda channel, then upload this package, the channeldata.json looks like this:

{
  "channeldata_version": 1,
  "packages": {
    "hello-inedo-triple": {
      "activate.d": false,
      "binary_prefix": false,
      "deactivate.d": false,
      "description": "Hello this is an “example” description",
      "dev_url": null,
      "doc_source_url": null,
      "doc_url": null,
      "home": "https://forums.inedo.com/topic/3696/conda-channels-should-also-add-the-constrains-from-a-package-s-index-file-to-repodata-json",
      "icon_hash": null,
      "icon_url": null,
      "identifiers": null,
      "keywords": null,
      "license": "None",
      "post_link": false,
      "pre_link": false,
      "pre_unlink": false,
      "recipe_origin": null,
      "run_exports": {},
      "source_git_url": null,
      "source_url": null,
      "subdirs": [
        "noarch"
      ],
      "summary": "Example package for Inedo/ProGet developers to improve their conda channel functionality",
      "tags": null,
      "text_prefix": false,
      "timestamp": 1677572777750,
      "version": "1.0.0"
    }
  },
  "subdirs": [
    "noarch"
  ]
}

But if I create a channel with conda-build using this example package only, the JSON file looks like this:

{
  "channeldata_version": 1,
  "packages": {
    "hello-inedo-triple": {
      "activate.d": false,
      "binary_prefix": false,
      "deactivate.d": false,
      "description": "Hello this is an \u201cexample\u201d description",
      "dev_url": null,
      "doc_source_url": null,
      "doc_url": null,
      "home": "https://forums.inedo.com/topic/3696/conda-channels-should-also-add-the-constrains-from-a-package-s-index-file-to-repodata-json",
      "icon_hash": null,
      "icon_url": null,
      "identifiers": null,
      "keywords": null,
      "license": "None",
      "post_link": false,
      "pre_link": false,
      "pre_unlink": false,
      "recipe_origin": null,
      "run_exports": {},
      "source_git_url": null,
      "source_url": null,
      "subdirs": [
        "noarch"
      ],
      "summary": "Example package for Inedo/ProGet developers to improve their conda channel functionality",
      "tags": null,
      "text_prefix": false,
      "timestamp": 1677572777,
      "version": "1.0.0"
    }
  },
  "subdirs": [
    "noarch"
  ]
}

You can see that in the channel created by conda-build, the "special" quotes in the "description" key are encoded characters, whereas the ProGet feed just puts the characters themselves in the JSON output.

Both are fine and proper JSON, but the way conda handles things means that some special characters should be encoded before putting them in the channeldata.json file.

Actually, I think it is ProGet that "does a bit too much" here. If you open the example package and look at info/about.json - which is the file where the description data is pulled from I guess - you see that in the package, the special quotes are also encoded. ProGet however decodes them before generating channeldata.json, leading to parsing problems in Conda itself.

I'm curious; any news on this topic?

I tested the new version (local, brand-new installation) and now all corrections to version numbers are consistent. Thanks for the quick fix!

Thanks! I was checking database entries (in relation to my other topic) and noted that you are probably generation the repodata.json on the fly based on the packages in the database/repository. So I realized it would be more complex than just adding a file.

But indeed, I think constrains should be treated the same way as depends.

I'll watch this thread, looking forward to the next update!

I'm happy to help you improving the conda functionality! Given the conda functionality being very new, It is totally OK that there is room for improvement

I created a small zip file containing a conda channel that can be downloaded here:
https://nextcloud.marin.nl/index.php/s/adkyQ59XojQNEn8

And also a link to the example package:
https://nextcloud.marin.nl/index.php/s/mzmbHjEjszwGYKB

It contains a fully indexed conda channel (with only one package though....), but you can see the repodata.json in the noarch folder, it looks like this:

{
  "info": {
    "subdir": "noarch"
  },
  "packages": {
    "hello-inedo-1.0.0-1.tar.bz2": {
      "build": "1",
      "build_number": 1,
      "constrains": [
        "scipy=1.9.3"
      ],
      "depends": [
        "numpy 1.22.*"
      ],
      "license": "None",
      "md5": "b72976a9e6b36563ecc988b5cc033055",
      "name": "hello-inedo",
      "noarch": "python",
      "sha256": "4b3eee67a7dc4475d8a84e3c5fd5b48924c0e472b59e87167b67a8f673523707",
      "size": 5169,
      "subdir": "noarch",
      "timestamp": 1676390732144,
      "version": "1.0.0"
    }
  },
  "packages.conda": {},
  "removed": [],
  "repodata_version": 1
}

So alongside the depends key, there is the constrains key. Note that there is no second t because its a verb, not a noun. The key defines how the package hello_inedo constrains the user when he installs scipy later on. The info is pulled from the index.json file in a package's info directory (see noarch/hello-inedo-1.0.0-1.tar.bz2) in the same way as the depends key which you are already using in ProGet. For reference, this is the index.json file in the hello-inedo package:

{
  "arch": null,
  "build": "1",
  "build_number": 1,
  "constrains": [
    "scipy=1.9.3"
  ],
  "depends": [
    "numpy 1.22.*"
  ],
  "license": "None",
  "name": "hello-inedo",
  "noarch": "python",
  "platform": null,
  "subdir": "noarch",
  "timestamp": 1676390732144,
  "version": "1.0.0"
}

Furthermore, conda's documentation is not fully covering the more advanced topics unfortunately. Quite some things I had to figure out the hard way....

But what I think you should do, is:

1. get the constrains key from the index.json that is in a package
2. add the info in the package's entry in the repodata.json file
3. In the package page in the UI, you can add them under "Optional dependencies" if ProGet supports that.

Because constrains is just a list of optional dependencies.

I hope this helps!

I noted that ProGet's conda channels does not add the constrains key from packages index-data to their index.

Conda packages have a very useful feature called "run_constraints". These are in fact optional dependencies, and when a user installs them conda will install the version & build that is specified in a package's constrains key.

Basically, ProGet should take the constrains key from a package's index.json file and add it to the package records in the repodata.json files. In a similar way to what is already done for the depends key.

I can supply a example package if needed.

@atripp @gdivis Thank you very much, I will test the new version early next week then and let you know if all is working as expected.

As for repairing existing entries: That should be OK I think, its only a small number of packages that have problems. And with a print of the repository and some regex, It should be easy to identify those.

Great, thanks! Glad to be of help.

Meanwhile, another kind of version detail that gives issues:
https://peps.python.org/pep-0440/#pre-release-separators

I gave one of my own package the version 2023.2.beta4. According to the above, it should be acceptable to have pre-release separators and also write beta instead of just b. ProGet adjusts it to 2023.2b4 which is the correct normalization. Again, similar to the above, the download URL works if I change it back to 2023.2.beta4 although ProGet stores it as 2023.2b4 internally.

Actually for this one, the conda client is wrong as well: searching for 2023.2.beta4 does not work. The conda client itself does not correctly normalize pre-release versions, it seems. But that's not for you to worry about :)

Looking forward to the fix, I'll watch this thread and test it once available.

@atripp Thanks again, I'm happy to help a bit here.

I think it is a different bug indeed, may be some inconsistencies in ProGet's inner workings. The steps to reproduce:

1. Create new CONDA feed without connector (I was using private/internal). Let's name it conda_channel
2. Download this file: https://anaconda.org/anaconda/ca-certificates/2023.01.10/download/win-64/ca-certificates-2023.01.10-haa95532_0.tar.bz2
3. Upload to the feed we just created
4. Now note: WebUI of the feed says there are no packages (error 1)
5. Check this file: PROGET_BASE_URL/conda/conda_channel/win-64/repodata.json
6. Note that the keys under the packages structure have their leading zeros removed.
7. Try to download: PROGET_BASE_URL/conda/conda_channel/win-64/ca-certificates-2023.1.10-haa95532_0.tar.bz2 => error
8. Add leading zero to URL => PROGET_BASE_URL/conda/conda_channel/win-64/ca-certificates-2023.01.10-haa95532_0.tar.bz2 => works

The issue is that conda combines the URL up to win-64 with the package name. And although ProGet saves the file without the leading zero, the URL is not correctly resolved for some reason.

GUI issues:

• This page: PROGET_BASE_URL/feeds/conda_channel/ca-certificates/versions => gives 500 error
• On feed page: PROGET_BASE_URL/feeds/conda_channel => you cannot see the package

Conda issue:

• Fortunately, the conda client does not care too much about literal versioning and removal of leading zeros. If I say I want version "2023.01.10", but on the ProGet server it has "2023.1.10", it will properly resolve to the correct package entry.
• But if the endpoint/URL is not working, conda cannot download the package.
• The problem is that the endpoint that ProGet offers is inconsistent in some way with its storage/database. See steps 7 and 8:
  -- although it removes the leading zeros before storing, the database-entry still includes (it seems) the leading zero.
  -- The download endpoint only responds to the URL including the leading zero, but downloads the file without leading zero
  -- The file storage does not have the leading zero

To get conda client to work with minimal effort, I think the only fix is to make the endpoint work properly (i.e. endpoint without the leading zero should work, as used in step 7).

For the GUI, I don't know the details of course. However, it seems that some things/actions just store the raw/original version, while other parts are storing a normalized version.

Regarding PEP440 normalization: I think conda normally just stores stuff as-is and applies normalization "on-thefly" when sorting/comparing is needed, for instance.

Btw - just noticed another case where the same issue occurs. A package with version "4.3.0a" is changed to "4.3.0a0". Which is normalization according to PEP440 - although just "4.3.0a" is not strictly forbidden. But again, the URL to download the package (this time including the added zero) does not work, but if I change the URL (change to literal "4.3.0a") it will work.

@atripp Thanks for your quick replies.

Are you sure 2023.01.10 should work? Because it doesn't work with ProGet 2022.0.19. I installed the latest version of ProGet, created a new conda feed, uploaded ca-certificates 2023.01.10. However, it is not uploaded/posted correctly.

• In the GUI, the feed overview says that there are no local packages in the feed.
• Going from the main page to packages, I can see the newly uploaded package (version indicates 2023.01.10 - which it also did with ProGet 6.0.20). Clicking on the package link gives me a 404 error.

Also tested with ca-certificates 2022.07.19, same issue.

If I use conda search -c CHANNEL_URL ca-certificates I see the package popping up, with version 2023.1.10, so without the leading zero. The package URL also contains the version number without leading zero. But trying to download that URL directly, does not work. However - and this is interesting I think - if I change the URL to include the leading zero, the download works!

But after starting the download, ProGet tries to forward me to the URL without the leading zero, giving me "package does not exist". Yet, I was able to download the files (although the downloaded file misses the leading zero, i.e. it changed the version)

I would not care too much about the GUI normally. Unfortunately, conda wants to use the URL from the index which is wrong so conda cannot download the package.

Overall, it seems the issue was not entirely fixed. I hope you can look at it again, thanks!

ps. looking at PEP440 - I don't see it explicitly excluding/forbidding the use of leading zeros. Actually, it even uses a version 2014.04 as an example. Then looking at conda's package specs: https://docs.conda.io/projects/conda/en/latest/user-guide/concepts/pkg-specs.html, again there is the example 1996.07 and I don't see it excluding leading zeros either.... Am I missing something?

Came across this issue on YouTrack of which the state is "completed":
https://inedo.myjetbrains.com/youtrack/issue/PG-2220/FIX-Error-with-Conda-package-versions-that-have-leading-0s

However, it still does not work. I tested with Proget 2022.0.19, but the file is still renamed to "2023.1.10" while the ProGet packages page shows "2023.01.10".

I think that issue should be re-opened.

I wanted to upload the ca-certificates (2023.01.10) package to a conda feed. The upload does not give any errors. On the 'packages' page (right from ProGet main page), the package is shown. But clicking on it gives a 404 error.

I also tried the 2022.10.11 version, and that one did work. Next, the 2022.07.19 does not work.... But the 2022.3.29 version just works.

It seems ProGet does not support packages with a calendar versioning scheme with leading zeros for the month/minor number.

I checked the storage, and ProGet is actually renaming the packages! In the storage, I have ca-certificates-2023.1.10-haa95532_0.tar.bz2, while the original filename is ca-certificates-2023.01.10-haa95532_0.tar.bz2 (including the zero). Meanwhile, ProGet does store the leading zero in its database, looking at the "packages" page.

Summarizing: there is a mismatch between what ProGet is putting in it's database and what ProGet is putting in it's storage for conda package with calendar version scheme. In my opinion, ProGet should not/never change the version of packages, even if it is to remove leading zeros. I hope you agree

In any case, thanks for looking into this issue! I hope it can get fixed.

EDIT: We're currently (still) on ProGet 6.0.20, realised that there have been quite some new versions already. Will check the release notes first.

Thanks for the quick action and the extra information. Did not realise conda build was making the check itself. We're going to install the new version and give it a try. Happy to help and make Proget better

Proget puts too much information in the repodata.json files in the subdirs (noarch, win-64, linux-64). This causes problems if the summary entries contain certain (non-unicode) special characters.

Unfortunately, there is no workaround. Conda tries to parse the repodata.json file but simply fails.

To reproduce the error; try to download uvicorn-0.17.1-py39hcbf5309_0.tar.bz2 from the conda-forge channel and upload it to a Conda feed. Next, conda will not be able to use the channel anymore.

Below image shows the difference between the subdir repodata.json generated by conda build (left) on my local PC, and the one that is generated by Proget (right)

Awesome! Thanks for the quick reply!

We have set-up a conda feed using Proget 6.0.11. In the documentation, it is suggested to use conda 4.10 or newer. However, with both 4.11 and 4.12, the conda feed cannot be used to create environments and/or install packages: it throws an error saying the "current_repodata.json" file cannot be found on the server.

Conda looks for two repodata files (specified via repodata_fns config key) by default: "current_repodata.json" and "repodata.json". The second is available, but the first one is not provided/created by ProGet.

If found two ways to work around this issue:

• For "conda install" or "conda create", you can specify the "--repodata-fn" option and set that to "repodata.json". Conda will not look for the current_repodata.json file.
• Secondly, one could override the "repodata_fns" key from default by adding it to their own conda configuration file: conda config --add repodata_fns repodata.json

But it would be much better if the Proget server would just generate these files and make them available via http, as those workarounds are quite advanced. Namely, the conda index command also generates these files by default.