Still cannot push Windows Docker image to ProGet

niklas.nilsson_8133

It doesn't seem to be currently possible to push Windows containers to a Docker Registry. Or am I mistaken here?

Best,
Nik

atripp

Hi Nik, ProGet does indeed support pushing Windows containers.

What sort of errors are you receiving?

niklas.nilsson_8133

So, after some further testing, I am able to push a small image based on windows-server-nano to my repository. But when I try to push a larger image I get "unauthorized: authentication required" after a while. Some layers succeed in pushing, but when I get to a larger layer it seems to stall.

PS C:\Program Files> docker push servername/registryname/imagename:tagname
The push refers to repository [servername/registryname/imagename]
73db119f1423: Pushed
39017a61fb52: Pushed
246da8a4d4c9: Pushed
a114056875fb: Pushed
9f99dd0b742e: Pushed
495c9abcf140: Pushed
63dc895b2a27: Pushed
d3dc3d141977: Pushed
5b7901df17bd: Pushed
3e1a4631d6cf: Pushing [==================================================>] 13.15GB/13.15GB
7526e29ba0ea: Pushed
b3bc068f2de5: Pushed
1fd88f70cb45: Pushed
1c5d24ff49f1: Pushed
bfd396c19d59: Pushing [==================================================>] 1.081GB
aa6445a500cd: Pushing [==================================================>] 890.4MB
cb4a90dd60da: Skipped foreign layer
bf16baf8d810: Skipped foreign layer
unauthorized: authentication required

Worth noting also, that if I subsequently try to push a smaller image again (without performing "docker login"), the push succeeds.

Any idea why that might be?

Best,

atripp

I think this is related to the foreign layer. This is a known issue, it's actually not an "error", but "by design" of Docker.

https://medium.com/@amlys/solution-docker-push-fails-with-skipping-foreign-layer-a119e03bdf0d

I think this will fix it.

niklas.nilsson_8133

Indeed, foreign layers is skipped for Windows containers for licensing reasons. I actually get the same "Foreign layer skipped" message when pushing nano, but the push succeeds, so I have doubts that it is connected to this. But I will test and get back in a few. See:

The push refers to repository [servername/registryname/nano]
e286a695fd3a: Skipped foreign layer
bc06b22070cd: Skipped foreign layer
tagname: digest: sha256:25c9ca4a8429b4afeba7bcbe06f2651c8a709a9d0f58b152fc1a0e81ab5f33b1 size: 1098

niklas.nilsson_8133

Hello!

So I've finally had time to verify this, and yes, the large image fails to push but the small image succeeds to push, even with the allow-nondistributable-artifacts option set.

Any ideas on how to continue from here?

Best,
Nik

atripp

Hi Nik;

There's nothing in ProGet that would yield this behavior, and it's not behavior that others experience, so I think it must be something between ProGet and your workstation. Maybe proxy server, request filtering, etc.?

Just to check, you are using chunked uploads with Docker, right? It's been the default for years, but i guess it can still be disabled. The same intermediate server may be stripping out ProGet's support of chunks? I don't know -- https://github.com/docker/distribution/blob/master/docs/spec/api.md#chunked-upload

We've also had reports of strange errors happening on random blobs from one or two customers. They discovered it was the Request Filtering feature enabled in IIS. The error was different, and it had to do with blocking query strings with :’s in them. But maybe it's this?

hope this helps

niklas.nilsson_8133

Hi,

So there's no chance a push could time-out if it takes a very long time or something like that? And yes, using default settings for chunked uploads, etc. It seems to only happen with very large images that take multiple hours to push.

Best,
Nik

atripp

ni Nik,

There could certainly be a timeout... but if that were the case, it wouldn't be causing an "authentication" error.

You can check errors in ProGet, Admin > Diagnostic Center, perhaps something is logged? If not, then ProGet isn't making the error.

We have seen intermediate servers (proxy, traffic shaping, etc), terminate long-running connections and return a strange/inappropriate error (i.e. something other than Gateway Timeout) to the client. It's really hard to guess. You can install an intermediate tool, like Fiddler or Wireshark, to capture traffic from your workstation. Maybe the raw traffic will give you a better clue.

niklas.nilsson_8133

@atripp said in Still cannot push Windows Docker image to ProGet:

install an intermediate tool, like Fiddler or Wireshark, to capture traffic from your workstation. Maybe the raw traffic will give you a better clue.

Hey.

No errors that seem relevant in the Diagnostic Center. I'll try to set up Fiddler and see if I can go from there.