OCI Support for Assets

dan.brown_0128

Would it be possible for Asset Directories to be accessible via OCI? This could actually remedy one of our largest challenges with using assets with our deployment tool (Octopus Deploy), but would be usable for other tools.

Currently in order to select a 'package" and version in Octopus, it must be one of the following feed types -- so we can't use upack as an option:

As a result, we often create Nugets for deployment assets even though it may not be c# based. For example we receive patch sets from a third party vendor (they are zip archives of Java patches) and they can be quite large (multi gig). We're uploading them into assets, but in our CI process, creating a nuget package so they can be selected on the deployment side.

By using OCI, we could keep the patches in the Asset directory, and not have to repackage them into a nuget (saving duplicated storage costs). Then on deployment, we could simply pick the proper version of the asset files (eg patch v2024.1, v2024.4, v2024.10, v2025.1 etc) and the deployment tool can interact with Assets natively over OCI protocol

All in all this would give us a similar experience to Helm and Docker chart/image version selection (currently using Azure Container Registry for OCI compliance) but for generic assets.

stevedennis

Hi @dan-brown_0128,

I'm afraid that's not possible -- Asset Directories are essentially a web-based file system where as OCI Registries are basically Docker Registries. Based on the context of that screenshot, I'd be surprised if the OCI registry could be used for something other than a container image registry.

OCI/Docker Registries can technically store any kind of large binary object, and does so using a 256-bit digest hash (think a double-guid). These "blobs" have no names or extensions - some are "tagged" to make them human readable/accessible. These tags are not only mutable, but can be anything you'd like (v2024.1 or purple).

That said, OCI Registries are not a suitable storage for artifacts by any means. Packages are ideal (self-contained metadata) - but at least files have names/extensions that will give you a clue as to the provenance.

That said, might I suggest a different approach for deployment to explore down the line, as you're looking to improve/optimize things:
https://inedo.com/buildmaster/vs-octopus-deploy

Thanks,
Steve

dan.brown_0128

Hi @stevedennis -- Thanks for your reply. I would like however to make some clarifications on some points you've made.

OCI is actually used for more than just transporting Container images -- in fact, helm charts are typically pulled and queried over the OCI protocol when set up as a proper helm repository. Ultimately helm charts are just a collection of files in a tarball (tgz). It is the industry norm to use OCI to interact with these repositories and query/pull chart versions and data.

Beyond container images and Helm Charts, the Open Container Initiative is specifically "designed generically enough to be leveraged as a distribution mechanism for any type of content" (https://opencontainers.org/about/overview/) (emphasis added). While the OCI is predominately for container technology, they do expressly state it is also designed for other content distribution.

Implementing OCI support inside of Proget is ultimately an Inedo decision, and we respect that business decision. However, we hope you have all the facts before making that call, particularly that OCI is actively used across the artifact management segment. As a prime example, Microsoft, in their Azure Container Registry supports container images, Helm charts, SBOM, scan results, etc in their OCI compliant registry (https://learn.microsoft.com/en-us/azure/container-registry/container-registry-manage-artifact). JFrog also supports various artifacts, WebAssembly modules, tar files, etc in their product (https://jfrog.com/blog/oci-support-in-jfrog-artifactory/). Cloudsmith supports similar additional artifacts in their OCI compliant registries (https://cloudsmith.com/blog/securely-store-and-distribute-oci-artifacts).

While you did hint at the initial intention of OCI, the industry has seen benefits and done exactly what OCI wanted, "enable innovation and experimentation above and around it" (https://opencontainers.org/faq/#what-are-the-values-guiding-the-oci-projects-and-specifications).

You might check out this blog article too (https://www.loft.sh/blog/leveraging-generic-artifact-stores-with-oci-images-and-oras) that even provides an example of pushing/pulling generic files (with custom filenames) to an OCI registry. You are correct that registry entries can be tagged as desired and those tags can be set/modified (depending on approved user permissions), just as your custom upack packages can be 'repackaged' - keeping the same content but providing a new pointer label.

Lastly we are aware of the BuildMaster product and have reviewed it, but we do not have any plans to switch from OctopusDeploy in the near future.

stevedennis

Thanks for the additional feed back @dan-brown_0128!

We do have a thread about OCI Registries in ProGet, as they have been requested from time to time. It might be worth posting this there, too?

But our current take is that "OCI Registries are a poorly-designed solution in search of a misunderstood problem", and that they are technologically inferior to alternatives. Here's a quote from that post:

The main issue I have is that an OCI registry is tied to a hostname, not to a URL. This is not what users expect or want with ProGet -- we have feeds. Users want to proxy public content, promotion content across feeds, etc. None of this is possible in an OCI registry.

We got Docker working as a feed by "hijacking" the repository namespace to contain a feed name. Helm charts don't have namespaces, so this is a no go.

Personally I can't imagine how this is scalable. A lot of new Docker users (including me) are "shocked" that you have to include the "source" in the container name (e.g. proget.corp/my-docker-feed/my-group/my-app) -- I can't see how this could ever work to expand this to all deployment artifacts (e.g. my-artifacts.corp/my-group/my-app/service-assembly), especially considering how often everything is referenced in scripts and dependencies.

Anyways best to continue on that other thread if you'd like to keep the discussion going, we're always open to learning more :)

dan.brown_0128

Hey @stevedennis -

From that quote, I get the sense that the container feeds were implemented by trying to force the process into what was implemented for traditional file-based packages rather than treating container images as a different style of artifact. This would explain why it seems OCI isn't a direct drop in fit -- because other traditional package consumers (pypi, nuget, maven, etc) do not behave the same as OCI.

As for the scalability comment - I can see your point if you tried to cram traditional package types into OCI where the client doesn't support that. Could clients like nuget support OCI for downloading the nupkg files? Sure. But they haven't so far as I know.

On the shock that for container images that you have to provide the URL to access the image -- that's not all that different than traditional software packages. Just on those traditional package tools you're specifying that URL Prefix as part of the client's configuration. The client may add in some "middle parts" to that url. And actually you do run into that some with docker: If you take the common nginx-ingress image hosted on DockerHub, the image name is just nginx/nginx-ingress which works just fine in docker and kubernetes deployments. Secretly, that full image URL parses out to docker.io/library/nginx/nginx-ingress.

Last one -- The URL thing. Take a look at some other common registries, including the public GitHub container registry. For those, the URL is always ghcr.io but they then prefix the image with the User/Org (eg: https://github.com/blakeblackshear/frigate/pkgs/container/frigate/394110795?tag=0.15.1). Truthfully, Proget could do a similar approach: proget.corp/MyOciOrDockerFeed/image/path/here:1.2.3