Caching ProGet feeds in S3 and/or auto-creating local repository

[cyril]

Hi all,

I'm currently in the process of migrating between ProGet instances; we're moving from a couple monolithic feeds to narrower, team-managed feeds, as well as moving from locally stored artifacts to S3-backed storage...

Since we've got a large amount of build pipelines & deployment environments that need to be updated, we figured that we could proxy our "old" instance, using connectors + team-specific filters. This works for switching endpoints easily, without a big-bang migration. Our teams can then progressively update to push to the new instance, while old tags are pulled from the old one. This also saves us from pulling data we don't need to the new instance.

One issue has come up however; the cached packages seem to be stored directly on disk, instead of being pushed to the (configured) S3 storage on the feed. Is this intended, or can it be reconfigured?

A related thought, to make our migration more 'resilient' would also be to automatically create a local repository when pulling via a connector, if that could be an option. However the S3 issue still remains; we won't be able to keep all our images locally.

Would you have any pointers on dealing with this?

Thanks,
Cyril

[stevedennis]

Hi @cyril ,

Actually, "cached" packages are stored the exact same way "local" packages are. The only difference is a flag (Cached_Indicator) on the database entry.

If you're seeing package files on local disk, they're might be from a different feed or are older (i.e. before you used cloud-storage).

Thanks,
Steve

[cyril]

@stevedennis thanks for the quick reply.

Something seems to going wrong then... For example, I've got a feed (backed by a connector) proxying docker.io images; essentially the same flow I'd like to use for our internal images.

I've pulled images through it with S3 enabled since the beginning. I'm expecting to see some data in there;

I've tested the storage, and it seems to be happy;

It does seem to be storing manifests, but no actual images;

I have caching enabled on the connector;

Inspecting ProGet itself (using the Docker image); it seems to be storing everything locally in /var/proget/packages;

Any idea what might be going on?

[stevedennis]

Hi @cyril ,

I can't see the full configuration, but do you have "common blob storage" configured on the feed? That shouldn't be possible with cloud storage enabled, but it might be something you did via the API or another bug.

Common Blob Storage is not supported with cloud storage at this time; it's something on our roadmap.

Thanks,
Steve

[cyril]

@stevedennis I didn't explicitly enable it, but it does seem to be enabled on those feeds indeed.. I did indeed use the API / pgutil.

I get the following when trying to click 'change'.

Only seems to affect Docker feeds; I've got other types of feeds (e.g., Helm) that I also configured; those don't show the same error when opening the GUI to edit.

[stevedennis]

Hi @cyril ,

Ah, that'll do it! That configuration is definitely not supported, so you'll get some weird behavior. Are you able to resolve/change it via the API back? That must not have the check mechanism.

Only Docker feeds use Common Blob Storage, since they're the only ones with a bunch of weird share layer files :)

Thanks,
Steve