@stefan-hakansson_8938 No, we haven't had any more luck and have put the debian mirror back on the backburner for the time being

Unfortunately we cannot share our scripts due to intellectual property rights. The hint I can give though is to note the format of the index.json in the public dist and maintain that in your private dist.

Being able to mirror the NodeJS distribution listing natively would be convenient. Currently we're having to do it by running a script daily that syncs the nodeJS Dist with an assets directory in proget

To share with the community some things learned from a private support ticket -- There's currently a defect in the logic where if no packages were downloaded/uploaded to a debian type feed, the in-memory cache is never started. This leads to very long runtimes when running apt update since the response times for the /Release and /InRelease endpoints (from our experience) are at best 1 minute, at worst 10 minutes.

The workaround support gave was to manually download a package via http before attempting to use apt. From our experience, this doesn't persist overnight, so it would basically need done daily and truly is counterintuitive to having a package manager like apt.

Are there plans to replace the database calls with API functions to secure database controls and access?

Is anybody else attempting to use Proget's debian feeds to mirror the official mirror for use in isolated environments?

I am curious what sort of performance others are seeing in that use case. Our prior tests found that mirroring via Proget was substantially slower than the official mirrors.

I do get your point of how versioning/tags are a bit different between traditional package artifacts vs container images. It can definitely have good and bad effects.

In regards to tag-mutability, you did bring up one method, using the SHA Digest since that will always point to a specific object. If the object changes, its hash changes. But humans don't like hashes, so tags were created.

Looking around, it does look like some registries actually support immutable tags (Ex ECR, Harbor). Some tags make sense to be excluded from immutability - like latest since those are dynamic by intention.

Bringing this thread back alive some to link out to: https://forums.inedo.com/topic/5364/oci-support-for-assets/

Hey @stevedennis -

From that quote, I get the sense that the container feeds were implemented by trying to force the process into what was implemented for traditional file-based packages rather than treating container images as a different style of artifact. This would explain why it seems OCI isn't a direct drop in fit -- because other traditional package consumers (pypi, nuget, maven, etc) do not behave the same as OCI.

As for the scalability comment - I can see your point if you tried to cram traditional package types into OCI where the client doesn't support that. Could clients like nuget support OCI for downloading the nupkg files? Sure. But they haven't so far as I know.

On the shock that for container images that you have to provide the URL to access the image -- that's not all that different than traditional software packages. Just on those traditional package tools you're specifying that URL Prefix as part of the client's configuration. The client may add in some "middle parts" to that url. And actually you do run into that some with docker: If you take the common nginx-ingress image hosted on DockerHub, the image name is just nginx/nginx-ingress which works just fine in docker and kubernetes deployments. Secretly, that full image URL parses out to docker.io/library/nginx/nginx-ingress.

Last one -- The URL thing. Take a look at some other common registries, including the public GitHub container registry. For those, the URL is always ghcr.io but they then prefix the image with the User/Org (eg: https://github.com/blakeblackshear/frigate/pkgs/container/frigate/394110795?tag=0.15.1). Truthfully, Proget could do a similar approach: proget.corp/MyOciOrDockerFeed/image/path/here:1.2.3

Hi @stevedennis -- Thanks for your reply. I would like however to make some clarifications on some points you've made.

OCI is actually used for more than just transporting Container images -- in fact, helm charts are typically pulled and queried over the OCI protocol when set up as a proper helm repository. Ultimately helm charts are just a collection of files in a tarball (tgz). It is the industry norm to use OCI to interact with these repositories and query/pull chart versions and data.

Beyond container images and Helm Charts, the Open Container Initiative is specifically "designed generically enough to be leveraged as a distribution mechanism for any type of content" (https://opencontainers.org/about/overview/) (emphasis added). While the OCI is predominately for container technology, they do expressly state it is also designed for other content distribution.

Implementing OCI support inside of Proget is ultimately an Inedo decision, and we respect that business decision. However, we hope you have all the facts before making that call, particularly that OCI is actively used across the artifact management segment. As a prime example, Microsoft, in their Azure Container Registry supports container images, Helm charts, SBOM, scan results, etc in their OCI compliant registry (https://learn.microsoft.com/en-us/azure/container-registry/container-registry-manage-artifact). JFrog also supports various artifacts, WebAssembly modules, tar files, etc in their product (https://jfrog.com/blog/oci-support-in-jfrog-artifactory/). Cloudsmith supports similar additional artifacts in their OCI compliant registries (https://cloudsmith.com/blog/securely-store-and-distribute-oci-artifacts).

While you did hint at the initial intention of OCI, the industry has seen benefits and done exactly what OCI wanted, "enable innovation and experimentation above and around it" (https://opencontainers.org/faq/#what-are-the-values-guiding-the-oci-projects-and-specifications).

You might check out this blog article too (https://www.loft.sh/blog/leveraging-generic-artifact-stores-with-oci-images-and-oras) that even provides an example of pushing/pulling generic files (with custom filenames) to an OCI registry. You are correct that registry entries can be tagged as desired and those tags can be set/modified (depending on approved user permissions), just as your custom upack packages can be 'repackaged' - keeping the same content but providing a new pointer label.

Lastly we are aware of the BuildMaster product and have reviewed it, but we do not have any plans to switch from OctopusDeploy in the near future.

Have you found a solution to disabling users in Proget when their access has been removed, or updating their groups when updated on the backend (without needing re-login to the web app)? We're running into challenges with keeping API Key access in sync with the user's actual access since Proget's internal user database doesn't synchronize against our identity backend.

Would it be possible for Asset Directories to be accessible via OCI? This could actually remedy one of our largest challenges with using assets with our deployment tool (Octopus Deploy), but would be usable for other tools.

Currently in order to select a 'package" and version in Octopus, it must be one of the following feed types -- so we can't use upack as an option:

As a result, we often create Nugets for deployment assets even though it may not be c# based. For example we receive patch sets from a third party vendor (they are zip archives of Java patches) and they can be quite large (multi gig). We're uploading them into assets, but in our CI process, creating a nuget package so they can be selected on the deployment side.

By using OCI, we could keep the patches in the Asset directory, and not have to repackage them into a nuget (saving duplicated storage costs). Then on deployment, we could simply pick the proper version of the asset files (eg patch v2024.1, v2024.4, v2024.10, v2025.1 etc) and the deployment tool can interact with Assets natively over OCI protocol

All in all this would give us a similar experience to Helm and Docker chart/image version selection (currently using Azure Container Registry for OCI compliance) but for generic assets.

I hadn't realized that the docker and helm feeds weren't OCI compatible. We're currently using Azure Container Registry for our docker/helm storage, but have been considering changing that as our usage grows and we're needing to tackle the Vulnerability/License compliance management.

In order to use the Proget docker/helm feeds as a drop in replacement, we'd absolutely need these to be OCI compliant and usable over the OCI Protocols.

In theory, having a filtered feed and promoting from that would work -- at least it would accomplish the same result.

The initial creation of the filters, at least in our case, would be quite lengthy (we have ~2500 packages in our approved feed today). Also, how scalable is the feed filtering? I could see the risk of performance issues as the number of packages in the filter grows.

I need to double check our policy/procedure for one specific. If package X is approved at version Y, I am not certain that versions prior to version Y are inherited in the approval. I know future ones are, and I think prior ones would be, but need to verify.

To piggyback on this -- this has been an idea we've been interested in as well. As part of our corporate policies, generally once a package has been approved (at a name level), all subsequent versions are OK assuming there's no vulnerabilities or license issues. Denying a request for this is very rare.

By automating version promotion, it would allow developers access to newer versions of packages sooner, making access easier and devs will be more likely to upgrade.

We thought about doing this by filters, but managing that list would quickly get out of hand.

This isn't a critical functionality for us today, more of a nice to have.

The reason for blocking application packages like this, that have vulnerable dependencies, comes down to security governance. While yes, it can be inconvenient if a package that has worked before is now blocked, that does prevent us from introducing known vulnerabilities into our environment. In the event of an emergency deployment (ex prod rollback, etc), we could apply a temporary exemption to allow the package to still deploy -- after doing a risk assessment.

To use the log4j example again -- if we have an application that was built with a vulnerable version of log4j, nobody would want that package to get deployed again (while also remediating it anywhere that it was already deployed). From what I can tell the most effective way would be to block the download from ProGet - if we can leverage it's automatic blocking functions.

Adding the audit into the deployment process is definitely one way to partly add this security layer but it'd require active implementation for all of our deployments, and opens more opportunity for teams to skip or work around it. Basically its better than nothing yes, but it's not the most effective security enforcement measure.

For #3 (which unfortunately is probably the biggest potential win for us), I might have an idea...

Currently when editing the Project's settings, you can specify Feeds - which I'm understanding are the feed(s) that provide the packages outlined in the SBOM - ie the dependency source from the build).

What if at that level it was broken out to input/output feeds? The artifact MyCorp.Package, which resulted in the SBOM, was stored in the "internal-apps" feed, and it was built using dependencies in the "approved-nuget-feed." Now, this would assume that the project name and resultant package name are the same, but that shouldn't be too unreasonable.

Again at a high level, what we're wanting is to be able to block our custom built packages if its dependencies (identified by SBOM) become vulnerable (or are otherwise deemed noncompliant). Since these packages will never be known by any OSS index or the ProGet Security, it wouldn't be flagged by traditional vulnerability means, and that's why the SCA/SBOM is useful.

In re-evaluating the SCA Functions of Proget 2024, we've come up with a few functionalities that would be big benefits. Posting it publicly in case others have similar thoughts

1. Be able to add a link to our CI build and CD deployment on a build's release (for Inedo, this was mentioned in EDO-10735)
   This could probably be as simple as adding two optional fields in the "Create Build" modal and/or in the build promotion process. Of course this would also want/need to be implemented into the pguitl CLI.

2. Dependent Package's 'Usage in Projects' should show latest version in addition to prior versions (EDO-10736)
   It'd be helpful to quickly see all versions of an SCA project that reference a specific version of a dependent package. Right now it's labeled for the Latest version, although on our instance it appears to not correctly show the latest.

3. Link between SCA Build and the artifact
   Bringing back up this topic from last year. Having a way to link an SCA Project/build back to the artifact that corresponds to the full BOM would be great, especially if it shows the vulnerabilities of the package discovered from the BOM analysis. This would let us block downloading of packages we've built that may now contain vulnerabilities that have come up over time.
   I wrote up a more narritive version of what I'm thinking:

• Imagine we have MyCorp.Package v1.0.0 which is a nuget stored in the internal-apps feed
• It has dependencies of Ext.Dep.1.1.1 and Ext.MS.Dep2.2.2 which are in the approved-nuget feed
• We upload the SBOM
• If you go to Ext.Dep or Ext.MS.Dep in approved-nuget feed you will see that each is used in the MyCorp.Package project (makes sense, those are dependencies of the package).
• But if you go to MyCorp.Package in internal-apps, you don't see it used in a project, since its not a dependency, but rather is the project itself. Wouldn't it make sense for this to link? That way if your package has a vulnerable dep (and the project shows vulnerabilities), you can see that at the top level (and maybe block downloading the applications package)t at the top level (and maybe block downloading the applications package)

I've noticed that other Artifact management systems are offering products for IaC Scanning -- looking for vulnerabilities, known misconfigurations, best practices, etc. This could be a feature that would be a nice supplement to ProGet.

Most of what I've seen has the IaC scanning centered around Terraform templates. I'm using ARM/Bicep since we're completely Azure based for cloud presence.

I'm not 100% sure how the other products are doing IaC scanning -- it may be along the lines of scanning a Terraform Private Registry, ensuring that modules there are up to standards. With Bicep/ARM, Azure provides Template Specs as a manner of storing and cataloging private modules that can be imported into templates. I don't think its prudent to reinvent Template Specs into ProGet, but it would be nice to get the best practice and quality checks to be made against ARM/Bicep templates stored inside of artifacts in ProGet.

In our instance, the templates are typically stored inside of a nuget package, although a universal package could be used instead.

Are there any others looking into IaC scanning, particularly in an Azure + ARM/Bicep organization?

I wonder if its an issue if the old JSONPath was pulled locally first, then jsonpath was pulled. Right now, we just have a workaround by disabling npm audit support in ProGet, which allows our builds to run, but that disables a legitimate feature because of this one edge case.

My guess is that some DB cleanup would need to happen to purge JSONPath and jsonpath from all memory?