hi all, thanks for the interest/comments; I decided to write-up a page that details this on the docs.

http://inedo.com/support/documentation/proget/feeds/other-types

I'm hoping we can use this public thread to maintain the discussion on technical detail; otherwise it'll get stuck in my email, or somewhere else, and we can get everyone to chime in this way.

That said, @M-W if you've got any insight into how R/CRAN works please do share :)

Of course, plan versioning is introduced in a later version, and Rafts (which allow for Git-based storage) is coming in 6.2.

I definitely understand the hesitance to upgrade, but worth noting ---- for once 6.2 comes around we'll offer some great migration tactics to let you pull applications from really old versions of BuildMaster. We'll also better support multi-instances of BuildMaster, so each group can upgrade as they'd like.

That said, the Plans database table will contain your OtterScript plans. Not sure if that's a good solution, but pulling from that table will give you latest versions.

Hello;

There may have been a miscommunication somewhere; do you know specifically what you were told?

We recently added the Feeds Management API, but that's only to manage feeds (not packages).

I just updated the documentation, and it will be published soon.

You can delete (permanently remove) or unlist (hide from most search results) NuGet packages from your feed by navigating to the package page and clicking the corresponding Delete Package or Unlist Package button. These actions require the Feeds_DeletePackage or Feeds_UnlistPackage permission attribute, respectively.

To programmatically delete a package from your feed, you can use the NuGet CLI's delete command, or make a DELETE request via HTTP:

DELETE http://{proget-server}/nuget/{feed-name}/package/{ID}/{VERSION}`

Note that this behavior is different than NuGet.org's DELETE command, which unlists packages instead.

To programmatically unlist (or relist) a package, you can use the NuGetPackagesV2_SetListed method within the Native API.

Is that helpful?

Definitely sounds like a bug; we've get it logged.

By the way, we've got some real exciting container/registry features coming very soon in ProGet 5.3. Stay tuned!

Hi @jeff-miles_5073 ,

Thanks for the first inquiry; I just updated the documentation! These must be "relatively new", and I know we've had a few customers using universal packages for this.

We have some Terraform integrations on our roadmap for 2023, so this definitely something we'll look into on our own as well.

Cheers,
Alex

Just as an update, we will be doing this:

https://inedo.myjetbrains.com/youtrack/issue/PG-1555

"Proxy npm audit requests to npmjs.org (experimental)"

Hi @shfunke_1795 , @jrottmann_6111

Thanks, I just added this to our documentation page!

I didn't look too deeply, but I found some initial documentation:
https://wiki.alpinelinux.org/wiki/Package_management

It seems this is "like Yum/RPM but for Alpine Linux"? None of us here use Alpine Linux, so there's a pretty big learning curve to get started. Any help here would be appreciated, and definitely move this along :)

Is this related to "APK" that Android uses?

Is the "API" mostly like basic file downloads, based on an index file? Are you able to "hack" or do a PoC using ProGet Asset Directories?

Hi @tkolb_7784

I wanted to provide an official answer to this:

Are there plans for (easy) HTTPS support in the internal webserver? Applying a self hosted certificate and deploying it to all build servers and developer machines seems over the top for me right now.

Yes. Not exactly sure how yet, but we would like the integrated webserver to support this as easily as possible.

@shayde @sebastian really appreciate the help, we'll get this incorporated ASAP !!

Thanks for clarifying @sebastian

So I'm not exactly thrilled by this UI, but maybe this is fine.

What do you think?

This is a kind of "quick and dirty" page that would show up if you clicked on that GPL-2.0 license and the "# projects" number.

Here's one for the packages as well:

Hey @dan-brown_0128,

And of course, the other issue with using SHA Digests is that they are unsortable/unordered, and there's no way to know if you're on an "newer" or "older" version. Git inherently has this same problem -- and users who try to name packages after commit hashes quickly learn how poorly that works :)

Looking around, it does look like some registries actually support immutable tags (Ex ECR, Harbor). Some tags make sense to be excluded from immutability - like latest since those are dynamic by intention.

ProGet's tags are effectively immutable -- in that you need the overwrite permission. But we also have the Semantic Container Versioning, which takes it a step further and enforces tags. Unfortunately, a lot of users don't use this feature or follow our guidance - and they end up with all the predictable headaches.

As an engineer, I'm obviously offended in principle by these bad practices, but more practically... the consequences of this kind of misuse/abuse is Tool Blame. And that means we look bad and don't get a renewal.

So, my philosophy/approach as a product vendor is "provide a good solution", not necessary "enough rope to hang yourself". As I mentioned before, OCI Repositories are poorly-designed solution to a misunderstood problem -- and my read is that it's going to be a passing fad.

Cheers,

Alex

Hey @davidroberts63,

Sure - let me elaborate on that.

A key tenet of software versioning is that a "version" represents a single, static/fixed revision of software - like there is only one "MyProduct 4.2.8", and no matter where/when you download that version, it's always the same thing.

OCI/Docker repositories do not have versions. It's just not a concept in the model, period. Instead, container images are "tagged," and it is up to the publisher to determine which container images have which tags and when those tags change.

This is all by design, and there's nothing inherently "wrong" with updating "MyProduct:4.2.8" until it's the right "version" of the container. If you want a "specific version of a version" you need to use the that utterly useless digest code. That's how these repositories are designed and used.

Compare this to package repositories, which are write-once by design and deletes are exceptional. This holds to the "a version is a static/fixed revision" principle.

ProGet does have the Semantic Versioning for Containers but it's not very popular, because it doesn't follow industry norm and developers want more control and customization over this.

Replacing/overwriting tags is the norm in a lot of container-based workflows, which causes all the problems you would expect.

All this is to say, I just can't see how OCI registries for generic artifacts are anything more than a poorly-designed solution in search of a misunderstood problem.

Cheers,
Alex

Hi @dan-brown_0128,

Thanks for continuing the discussion.

From that quote, I get the sense that the container feeds were implemented by trying to force the process into what was implemented for traditional file-based packages rather than treating container images as a different style of artifact. This would explain why it seems OCI isn't a direct drop in fit -- because other traditional package consumers (pypi, nuget, maven, etc) do not behave the same as OCI.

Docker and Asset feeds are implemented considerably different than the other feeds, and even have a different UI/pages because of that. But the model of "feeds" and "proxying" (i.e. what people want to do with ProGet) is just fundamentally incompatible.

The OCI API is effectively identical to Docker API, but there's no sensible way to display/visualize "generic" artifacts. They have no name, just useless digest hashes. It'd be like using Git without comments, dates, or files names.

On the shock that for container images that you have to provide the URL to access the image -- that's not all that different than traditional software packages. Just on those traditional package tools you're specifying that URL Prefix as part of the client's configuration.

That's a pretty big deal IMO, and why this approach is dead in the water when dependencies are involved. It's a huge headache with Helm already - URL Prefixs need to change.

It's bad enough every single one of your build/deployment scripts need to be updated if you want to change the "URL Prefix" of a "source" (switching products, domain name changes, etc.) -- but these URL Prefixes are embedded in manifest/metadata files stored in the registry.

Of course, you can't "edit" those files stored in the registry, since the digest/hash would change, so you have to republish everything... which would cause anything that references those digests/hash to break.

If you've "transcended versioning" and always deploy the latest commit of everything always, then none of this really matters - but versioning is important for a lot of users still. Tags are not suitable for versioning.

Last one -- The URL thing. Take a look at some other common registries, including the public GitHub container registry. For those, the URL is always ghcr.io but they then prefix the image with the User/Org (eg: https://github.com/blakeblackshear/frigate/pkgs/container/frigate/394110795?tag=0.15.1). Truthfully, Proget could do a similar approach: proget.corp/MyOciOrDockerFeed/image/path/here:1.2.3

This is what ProGet has to do now -- we "hijack" the first part of the container namespace for the feed name, and GitHub does that same thing. But you're basically stuck forever with that host/feed name.

Hey Michal,

Check out latest pgutil (2.1.7) - I just made a quick change and now Asset will work.

Also, so will Maven2.

Best,
Alex

hi @jacob-kemme_1710 , thanks for all of the details.

I'm pretty sure the Docker feeds already support OCI containers. They are basically identical from an API standpoint, and we've made a few tweaks here and there to accept different media types or parse a field in a manifest differently.

I know for sure that vnd.oci.image.manifest.v1+json is already supported.. and if there's something missing, it should be trivial to add. I'd post that as a separate feature request, since it's likely a basic tweak.

As for Helm... that's a different story. I don't think it's going to work, and it doesn't seem like a good fit for ProGet. Nor is a "generic OCI registry", which is why I'm hesitant to even consider the feature.

The main issue I have is that an OCI registry is tied to a hostname, not to a URL. This is not what users expect or want with ProGet -- we have feeds. Users want to proxy public content, promotion content across feeds, etc. None of this is possible in an OCI registry.

We got Docker working as a feed by "hijacking" the repository namespace to contain a feed name. Helm charts don't have namespaces, so this is a no go.

We got Terraform feeds working by requiring some stupid prefix on the module name. There's just no other way to do it.

I don't think that would be smart to do with Helm. And obviously we couldn't do that for "generic OCI" content since it's just meaningless blobs/binaries.

I'm open to learning more... but my initial take is that if users don't find values in "feeds" (well-defined content types, permissions, segmentation, replication, etc), and just want a "generic place to shove random content stuff" then ProGet isn't the tool?

Alex

@dan-brown_0128 thanks for clarifying

2500 is a lot more than I'm thinking... I was envisioning a handful of frequently-updated / highly-trusted packages (like AWSSDK, ). I think the UI would struggle a bit.

Thinking about this further... a blanket "all future versions are OK" policy seems like it will cause more problems than benefits. That's a lot of decision-making burden to shift to developers, and they're just going to hit "update" the moment they're prompted.

Best case scenario, that update delivers Zero Business Value... but there's a chance it'll introduce a regression, etc. I'd really need to study the data before recommending a practice like this (let adding a supported usecase for it) I'd be open to some sort of "automatic approval" feature, but I'd want to see some other factors in there, other than just "name matches X".

As it stands, "package approval/promotion" isn't very widely used as it's solving a problem most organizations don't believe they have. Instead they drop $300K+ for Clownstrike or another security tool and believe it just "does everything for them". At that price, how could it not!

So I think it's best to pin this for a bit. It's still a pretty "frontier" discussion/feature, and I don't want to "invent" something that's not going to get a lot of widespread use.

Hi @dan-brown_0128, @scampbell_8969,

We were reviewing this as part of our ProGet 2025+ roadmap, but I'm not sure if it makes sense to do. To summarize...

"Once a package has been approved (at a name level), all subsequent versions are OK assuming there's no vulnerabilities or license issues. However, the current promotion model requires that we promote each and every version. By automating version promotion, it would allow developers access to newer versions of packages sooner, making access easier and devs will be more likely to upgrade."

What about just using a connector with package filters in this case? For example:

1. Create feed nuget-unapproved, nuget-approved
2. Restrict promotion from nuget-unapproved -> nuget-approved
3. Create connector NuGet.org-all and associate with nuget-unapproved
4. Create connector NuGet.org-approved and associate with nuget-approved
5. Edit connector filters on NuGet.org-approved to block everything except packages you want

This would be possible today and would avoid adding a complex feature

Thanks,
Alex

Thanks @steviecoaster! All that sounds great, especially since it won't require changing anything on our end :)

Much appreciateD!!

@steviecoaster great thanks! That sounds good to me, I think having a new install option would be great.

Aside from supporting the package (which I'm not really worried about since you built it!!), the main concern I would have is keeping up with versions. We have frequent ProGet releases and pgutil is basically on demand.

I vaguely remember there was some kind of auto-packaging thing? Or maybe I'm dreaming that?

We could add something to our deployment plan that "does something" as well.

Hi @steviecoaster ,

That's awesome! I just added steviecoaster as a maintainer of the package, and it says pending approval.

Quasi-related, but I delisted all versions of romp and upack since we no longer publish/support those tools, but they are still showing up: https://community.chocolatey.org/profiles/inedo

Not sure if it takes time to reindex or whatnot?

Cheers,
Alex