Caching ProGet feeds in S3 and/or auto-creating local repository

[cyril]

Hi all,

I'm currently in the process of migrating between ProGet instances; we're moving from a couple monolithic feeds to narrower, team-managed feeds, as well as moving from locally stored artifacts to S3-backed storage...

Since we've got a large amount of build pipelines & deployment environments that need to be updated, we figured that we could proxy our "old" instance, using connectors + team-specific filters. This works for switching endpoints easily, without a big-bang migration. Our teams can then progressively update to push to the new instance, while old tags are pulled from the old one. This also saves us from pulling data we don't need to the new instance.

One issue has come up however; the cached packages seem to be stored directly on disk, instead of being pushed to the (configured) S3 storage on the feed. Is this intended, or can it be reconfigured?

A related thought, to make our migration more 'resilient' would also be to automatically create a local repository when pulling via a connector, if that could be an option. However the S3 issue still remains; we won't be able to keep all our images locally.

Would you have any pointers on dealing with this?

Thanks,
Cyril

[stevedennis]

Hi @cyril ,

Actually, "cached" packages are stored the exact same way "local" packages are. The only difference is a flag (Cached_Indicator) on the database entry.

If you're seeing package files on local disk, they're might be from a different feed or are older (i.e. before you used cloud-storage).

Thanks,
Steve

[cyril]

@stevedennis thanks for the quick reply.

Something seems to going wrong then... For example, I've got a feed (backed by a connector) proxying docker.io images; essentially the same flow I'd like to use for our internal images.

I've pulled images through it with S3 enabled since the beginning. I'm expecting to see some data in there;

I've tested the storage, and it seems to be happy;

It does seem to be storing manifests, but no actual images;

I have caching enabled on the connector;

Inspecting ProGet itself (using the Docker image); it seems to be storing everything locally in /var/proget/packages;

Any idea what might be going on?