Docker registry mirroring support

[luminiscence_6236]

Vanilla docker registry supports mirroring with pull-through cache feature. When user tries to pull some image hosted on dockerhub, docker daemon redirects this request to mirror, mirror checks if image is cached locally, and if not, fetches its layers and manifests and responds with cached local content.
I am wondering is this feature implemented in ProGet in some way. Docker registry connector, mentioned here and here didn't work for me, because registry address from vanilla docker registry example (https://registry-1.docker.io/v2) requires bearer authentication and ProGet fails with "authentication required" error. Even when I got it to work with some another address, it just fetched all manifests from dockerhub (what is quite unexpected behavior), but I'm not sure is it really mirroring functionality I want, because it is quite hard to understand whether image was pulled from dockerhub directly or from ProGet.

[atripp]

Hello;

Sounds like there's a lot to unpack...

First, we've got some major changes in ProGet 5.3 that should help out quite a bit, but I've included a lot of resources in this post to consider as well. Consider that, as a private registry, ProGet is designed to host "private images" that you manage (which can be build from "public images"), and it's really not designed as a "registry mirror/cache". These are two different usecases.

The authentication issues, that might be different problem and might be worth a new thread?

Repository Mirroring in ProGet

From But here is our current thoughts/documentation on repository mirroring in Proget:

There is the ability to "mirror" hub.docker.org, but this feature was really only designed to enable Docker, Inc. to host public registries in sensitive regions like China (so you can mirror to registry.docker-cn.com instead).
Unfortunately, the Docker client will only use the configured "mirror" for building and pulling; images are always pushed to hub.docker.org.
Docker, Inc. does not seem to be interested in enabling this, and we aren't particularly keen on forking the Docker client to enable this. Thus, it seems there's no sense in ProGet supporting this anytime soon

Base Image Pattern

On Dockerhub, there are a lot of problematic images that cause a new breed of "worked on my Docker" problems in companies. openjdk:11 is a great example of this problem. The connector "problems" in ProGet are a symptom of the problem, but we strongly recommend you to not use "third-party" container image directly, but instead follow our "base image" development pattern.

First, and foremost, openjdk:11 currently references what's called a "fat" manifest, which is a sort of "pointer" to other manifests that will be used depending on the run-time environment of Docker. There are currently four different environment-specific images that this "fat" manifest points to, so when you build an image on top of it (e.g. FROM openjdk:11), your image will be "locked" to whatever environment configuration the build machine (or even workstation) had at the time the command was run. This may be different than the machine you run the image on.

Even more problematic, however, is that 11 is a time-sensitive tag, and what 11 refers to keeps changing. Today, it points to four different versions of four different images (as of 7 days ago), but tomorrow... it could point to 3 totally different images. Who knows? It's a repository maintained by a third party, and 11 means what they say it means, at their discretion.

The Dockerhub is meant to be the "authority" on this repository, and this is where connectors are a challenge. ProGet caches certain informations about these images, but when the meaning of these things change, it defeats the purpose of caching. If you disable caching, it should work... but then you've already , but what's the point then? You're downloading huge image files constantly then.

So instead, here's how we recommend you develop a practice around third-party base images.

docker pull openjdk:11
docker tag openjdk:11 proget.mycompany.com/dev-images/openjdk-linux64:11.0.6-rc.1
docker push proget.mycompany.com/devimages/mycomp/openjdk-linux64:11.0.6-rc.1

Even better would be to use a Dockerfile (e.g. FROM openjdk:11) that lets you customize the image as needed, but this a good start. Then, after you/developers are happy with that version of that image, then you can "re-tag" from 11.0.6-rc.1 to 11.0.6. In ProGet 5.3, these operations are made easier, and "virtual" tags are automatically generated. But the important thing is that you treat "tags" as immutable, and provide consistent images for your users.

Note that, the repository name is mycomp/openjdk-linux64. It's important to not use only openjdk-linux64, since that name is identical to _/openjdk-linux64 and library/openjdk-linux64. The library/empty/_ namespace are intended only for "Dockerhub-certified" images

Docker is supposed to solve "works on my machine", but unless you treat tags as immutable, you'll keep getting "works on my Docker when I ran it," and that's even harder to debug. ProGet is "exposing" the underlying problems with these machine-specific, time-specific tags, but I guess it's better to discover it early on, then in Production.