OCI support?

edward.a.peng_7759 · 23 May 2024, 15:32

Hi,
Does the current docker feed support OCI (open container initiative) specification similar to dockerhub? or is there a plan to implement a separate OCI feed type?

if neither is true, is it possible to consider having proget to support OCI in the near future? With Helm v3.8 GA'd OCI support few years back, most charts are moving towards storing in OCI registry instead of the older helm repository. most hosted registry solutions support this for quite some time now (https://helm.sh/docs/topics/registries/#use-hosted-registries)

atripp · 24 May 2024, 02:13

Hi @edward-a-peng_7759 ,

There are no plans for a "generic" OCI Registry feed type. However, if you can help us understand what value this could add, we can consider building one.

A generic OCI registry seems to be like a "dumb" file system that's built around "dumb" cloud storage like S3, etc. What I mean, it's just files with no context. There is no real visibility into what's stored an OCI registry - it's just a place to store and access unnamed files via digests.

ProGet is a "smart" package and container system, and stores OCI-based container images (i.e. Docker images) in a Docker feed, and Helm charts in a Helm feed. There are so many advantages to this "smart" system vs a "dumb" registry:

You can browse images and charts in the UI
Charts and images are "linked" and you can see what charts reference which images
You can see what packages are used by images
Vulnerabilities are visible in charts, which packages/containers use, etc.
You can track downloads and configure permissions and control access

A "dumb" file system obviously couldn't offer any of this - and is one of the reasons people prefer ProGet over ECR, ACR, GCR, etc.

So far as I can tell, it doesn't matter to client tools whether you use ProGet Feeds or a generic OCI registry -- everything works the same with regards to helm, docker, kubernetes, and other supported tools.

Are there any advantages to having Helm charts being stored or accessed in a different manner?

Thanks,
Alana

jacob.kemme_1710 · 19 Feb 2025, 21:55

In our case, we're not specifically looking for Proget to add a feed type that is a generic implementation of OCI, we are asking for the existing feed types to fully support OCI image and feed standards where applicable (container images, helm charts, SBOMs, etc). At the very least, for container and helm feeds. The specific purpose of OCI is ensuring compatibility and standardization across different platforms and tools. It appears Proget is now behind nearly every other product in this regard for these artifact types. Proget at the very least should support and preserve OCI schema elements related to each feed type:

Proget Docker feeds should accept and preserve oci schema elements, example:

artifactType on descriptors if supplied (introduced in 2023)
mediaType(s): application/vnd.docker.distribution.manifest.v2+json as well as OCI Image Media Types

Proget helmchart feeds should should accept and preserve oci schema elements, example:

artifactType: application/vnd.cncf.helm.chart if supplied
application/tar+gzip when using index.yaml for classic/ existing charts
Helm OCI MediaTypes (used by default in Helm as of 3.8), or convert to OCI entirely since all of Helm is now on ArtifactHub.io (already supported by Proget as of 2024.17)

Notable container registries already supporting the Open Container Initiative image and distribution specs:

Azure Container Registry (ACR) added support 2023
Google Container Registry (GCR) transitioned to Google Artifact Registry (ACR) 2021-2024
Microsoft Container Registry (MCR) added support 2023
Amazon ECR added support 2020
GitLab Container Registry added support 2021
Artifactory added support 2023
OpenRegistry added support 2023
Docker Hub added support 2023
Zot (2020 OCI-only)

Docker desktop builds vnd.oci.image.manifest.v1+json by default now, not vnd.docker.distribution.manifest.v2
Zot only supports vnd.oci.image.manifest
OCI supports zstd and zstd:chunked. Not only is this a better compression format, which should speed up image pulls, zstd:chunked will allow container tools like Podman, Buildah and CRI-O to pull only the files that have changed in an image rather than the entire image.
Buildah, a Red Hat tool, allows building (only OCI) container images without docker daemon as a requirement

Beyond this subset of examples, it's becoming increasingly harder to find tools related to this space that haven't implemented OCI standards. Without OCI compliance in Proget container/ helm feeds, we're forced to look at other feed solutions that are keeping up with industry standards for proper tool interoperability. It seems like it would be a large win for Proget to implement OCI as a base, and extend that into the container (docker v2 images+feed schema are already OCI compatible), chart, sbom, and generic/ a (new) oci artifact feed type based on known OCI artifactType/mediaType definitions.

dan.brown_0128 · 20 Feb 2025, 19:08

I hadn't realized that the docker and helm feeds weren't OCI compatible. We're currently using Azure Container Registry for our docker/helm storage, but have been considering changing that as our usage grows and we're needing to tackle the Vulnerability/License compliance management.

In order to use the Proget docker/helm feeds as a drop in replacement, we'd absolutely need these to be OCI compliant and usable over the OCI Protocols.

apxltd · 21 Feb 2025, 14:04

hi @jacob-kemme_1710 , thanks for all of the details.

I'm pretty sure the Docker feeds already support OCI containers. They are basically identical from an API standpoint, and we've made a few tweaks here and there to accept different media types or parse a field in a manifest differently.

I know for sure that vnd.oci.image.manifest.v1+json is already supported.. and if there's something missing, it should be trivial to add. I'd post that as a separate feature request, since it's likely a basic tweak.

As for Helm... that's a different story. I don't think it's going to work, and it doesn't seem like a good fit for ProGet. Nor is a "generic OCI registry", which is why I'm hesitant to even consider the feature.

The main issue I have is that an OCI registry is tied to a hostname, not to a URL. This is not what users expect or want with ProGet -- we have feeds. Users want to proxy public content, promotion content across feeds, etc. None of this is possible in an OCI registry.

We got Docker working as a feed by "hijacking" the repository namespace to contain a feed name. Helm charts don't have namespaces, so this is a no go.

We got Terraform feeds working by requiring some stupid prefix on the module name. There's just no other way to do it.

I don't think that would be smart to do with Helm. And obviously we couldn't do that for "generic OCI" content since it's just meaningless blobs/binaries.

I'm open to learning more... but my initial take is that if users don't find values in "feeds" (well-defined content types, permissions, segmentation, replication, etc), and just want a "generic place to shove random content stuff" then ProGet isn't the tool?

Alex

dan.brown_0128

Bringing this thread back alive some to link out to: https://forums.inedo.com/topic/5364/oci-support-for-assets/

apxltd

Hi @dan-brown_0128,

Thanks for continuing the discussion.

From that quote, I get the sense that the container feeds were implemented by trying to force the process into what was implemented for traditional file-based packages rather than treating container images as a different style of artifact. This would explain why it seems OCI isn't a direct drop in fit -- because other traditional package consumers (pypi, nuget, maven, etc) do not behave the same as OCI.

Docker and Asset feeds are implemented considerably different than the other feeds, and even have a different UI/pages because of that. But the model of "feeds" and "proxying" (i.e. what people want to do with ProGet) is just fundamentally incompatible.

The OCI API is effectively identical to Docker API, but there's no sensible way to display/visualize "generic" artifacts. They have no name, just useless digest hashes. It'd be like using Git without comments, dates, or files names.

On the shock that for container images that you have to provide the URL to access the image -- that's not all that different than traditional software packages. Just on those traditional package tools you're specifying that URL Prefix as part of the client's configuration.

That's a pretty big deal IMO, and why this approach is dead in the water when dependencies are involved. It's a huge headache with Helm already - URL Prefixs need to change.

It's bad enough every single one of your build/deployment scripts need to be updated if you want to change the "URL Prefix" of a "source" (switching products, domain name changes, etc.) -- but these URL Prefixes are embedded in manifest/metadata files stored in the registry.

Of course, you can't "edit" those files stored in the registry, since the digest/hash would change, so you have to republish everything... which would cause anything that references those digests/hash to break.

If you've "transcended versioning" and always deploy the latest commit of everything always, then none of this really matters - but versioning is important for a lot of users still. Tags are not suitable for versioning.

Last one -- The URL thing. Take a look at some other common registries, including the public GitHub container registry. For those, the URL is always ghcr.io but they then prefix the image with the User/Org (eg: https://github.com/blakeblackshear/frigate/pkgs/container/frigate/394110795?tag=0.15.1). Truthfully, Proget could do a similar approach: proget.corp/MyOciOrDockerFeed/image/path/here:1.2.3

This is what ProGet has to do now -- we "hijack" the first part of the container namespace for the feed name, and GitHub does that same thing. But you're basically stuck forever with that host/feed name.

davidroberts63

I agree the hostname aspect is problematic at times. We've seen that with npm lock files disrupting build processes because our build servers block access to the public npm registry.

Would you elaborate on what you meant by "Tags are not suitable for versioning."? As far I am aware, tags are the primary way to denote different versions of a container image. Yes, the 'latest' tag is a shortcut to the most recent build of an image. And as you mentioned, many users consider specific versioning to be important, we are one of those users. But if tags are not for versioning container images, what is?

apxltd

Hey @davidroberts63,

Sure - let me elaborate on that.

A key tenet of software versioning is that a "version" represents a single, static/fixed revision of software - like there is only one "MyProduct 4.2.8", and no matter where/when you download that version, it's always the same thing.

OCI/Docker repositories do not have versions. It's just not a concept in the model, period. Instead, container images are "tagged," and it is up to the publisher to determine which container images have which tags and when those tags change.

This is all by design, and there's nothing inherently "wrong" with updating "MyProduct:4.2.8" until it's the right "version" of the container. If you want a "specific version of a version" you need to use the that utterly useless digest code. That's how these repositories are designed and used.

Compare this to package repositories, which are write-once by design and deletes are exceptional. This holds to the "a version is a static/fixed revision" principle.

ProGet does have the Semantic Versioning for Containers but it's not very popular, because it doesn't follow industry norm and developers want more control and customization over this.

Replacing/overwriting tags is the norm in a lot of container-based workflows, which causes all the problems you would expect.

All this is to say, I just can't see how OCI registries for generic artifacts are anything more than a poorly-designed solution in search of a misunderstood problem.

Cheers,
Alex

dan.brown_0128

I do get your point of how versioning/tags are a bit different between traditional package artifacts vs container images. It can definitely have good and bad effects.

In regards to tag-mutability, you did bring up one method, using the SHA Digest since that will always point to a specific object. If the object changes, its hash changes. But humans don't like hashes, so tags were created.

Looking around, it does look like some registries actually support immutable tags (Ex ECR, Harbor). Some tags make sense to be excluded from immutability - like latest since those are dynamic by intention.

apxltd

Hey @dan-brown_0128,

And of course, the other issue with using SHA Digests is that they are unsortable/unordered, and there's no way to know if you're on an "newer" or "older" version. Git inherently has this same problem -- and users who try to name packages after commit hashes quickly learn how poorly that works :)

Looking around, it does look like some registries actually support immutable tags (Ex ECR, Harbor). Some tags make sense to be excluded from immutability - like latest since those are dynamic by intention.

ProGet's tags are effectively immutable -- in that you need the overwrite permission. But we also have the Semantic Container Versioning, which takes it a step further and enforces tags. Unfortunately, a lot of users don't use this feature or follow our guidance - and they end up with all the predictable headaches.

As an engineer, I'm obviously offended in principle by these bad practices, but more practically... the consequences of this kind of misuse/abuse is Tool Blame. And that means we look bad and don't get a renewal.

So, my philosophy/approach as a product vendor is "provide a good solution", not necessary "enough rope to hang yourself". As I mentioned before, OCI Repositories are poorly-designed solution to a misunderstood problem -- and my read is that it's going to be a passing fad.

Cheers,

Alex