Unable to push images to self hosted Proget

mhdos_4222

We have deployed a free edition of Proget 2022.19 (Build 10) on our self hosted K8s. The K8s cluster runs on ec2 instances hosted in AWS and is exposed via an classic ELB on AWS to an https endpoint. The WebBase.URL is also set to the URL of the self-hosted Proget.

Until last week, we were able to push images and pull from our self hosted Proget registry without any issues. However, since last Friday, we are unable to push or pull images from the registry. We attempted with a fresh installation after clearing the persistent volume behind MSSQL and Proget, but the results were the same.

The K8s cluster is of version 1.21.4 and uses containerd 1.7.0 as the container runtime.

Given below is the debug output of the push command. I can see several Unauthorized entries, but unsure why they are there. The push command was executed after successfully logging into our self hosted Proget.

Any ideas on what is happening?

ubuntu@ip-10-0-2-68:~/proget$ sudo nerdctl --debug-full image push self-hosted-proget.dropthought.com/dt-onprem/ubuntu:bionic
DEBU[0000] converted                                     new="<nil>" old="{application/vnd.oci.image.layer.v1.tar+gzip sha256:0c5227665c11379f79e9da3d3e4f1724f9316b87d259ac0131628ca1b923a392 25688211 [] map[] [] <nil> }"
DEBU[0000] converted                                     new="<nil>" old="{application/vnd.oci.image.config.v1+json sha256:3941d3b032a8168d53508410a67baad120a563df67a7959565a30a1cb2114731 2297 [] map[] [] <nil> }"
DEBU[0000] converted                                     new="<nil>" old="{application/vnd.oci.image.manifest.v1+json sha256:0779371f96205678dbcaa3ef499be2e5f262c8b09aadc11754bf3daf9f35e03e 424 [] map[] [] 0xc00002be60 }"
DEBU[0000] converted                                     new="&{application/vnd.oci.image.index.v1+json sha256:cfa36a700a8c12f0f16a243188e0731dd81d875fc266eb69b98de669cede4a4d 289 [] map[] [] <nil> }" old="{application/vnd.oci.image.index.v1+json sha256:8aa9c2798215f99544d1ce7439ea9c3a6dfd82de607da1cec3a8a2fae005931b 1334 [] map[] [] <nil> }"
INFO[0000] pushing as a reduced-platform image (application/vnd.oci.image.index.v1+json, sha256:cfa36a700a8c12f0f16a243188e0731dd81d875fc266eb69b98de669cede4a4d)
DEBU[0000] Ignoring hosts dir "/etc/containerd/certs.d"  error="stat /etc/containerd/certs.d: no such file or directory"
DEBU[0000] Ignoring hosts dir "/etc/docker/certs.d"      error="stat /etc/docker/certs.d: no such file or directory"
DEBU[0000] pushing                                       digest="sha256:cfa36a700a8c12f0f16a243188e0731dd81d875fc266eb69b98de669cede4a4d" image="self-hosted-proget.dropthought.com/dt-onprem/ubuntu:bionic"
DEBU[0000] push                                          digest="sha256:0c5227665c11379f79e9da3d3e4f1724f9316b87d259ac0131628ca1b923a392" mediatype=application/vnd.oci.image.layer.v1.tar+gzip size=25688211
DEBU[0000] checking and pushing to                       digest="sha256:0c5227665c11379f79e9da3d3e4f1724f9316b87d259ac0131628ca1b923a392" mediatype=application/vnd.oci.image.layer.v1.tar+gzip size=25688211 url="https://self-hosted-proget.dropthought.com/v2/dt-onprem/ubuntu/blobs/sha256:0c5227665c11379f79e9da3d3e4f1724f9316b87d259ac0131628ca1b923a392"
DEBU[0000] do request                                    digest="sha256:0c5227665c11379f79e9da3d3e4f1724f9316b87d259ac0131628ca1b923a392" mediatype=application/vnd.oci.image.layer.v1.tar+gzip request.header.accept="application/vnd.oci.image.layer.v1.tar+gzip, */*" request.header.user-agent=containerd/1.7.0-beta.3+unknown request.method=HEAD size=25688211 url="https://self-hosted-proget.dropthought.com/v2/dt-onprem/ubuntu/blobs/sha256:0c5227665c11379f79e9da3d3e4f1724f9316b87d259ac0131628ca1b923a392"
DEBU[0000] push                                          digest="sha256:3941d3b032a8168d53508410a67baad120a563df67a7959565a30a1cb2114731" mediatype=application/vnd.oci.image.config.v1+json size=2297
DEBU[0000] checking and pushing to                       digest="sha256:3941d3b032a8168d53508410a67baad120a563df67a7959565a30a1cb2114731" mediatype=application/vnd.oci.image.config.v1+json size=2297 url="https://self-hosted-proget.dropthought.com/v2/dt-onprem/ubuntu/blobs/sha256:3941d3b032a8168d53508410a67baad120a563df67a7959565a30a1cb2114731"
DEBU[0000] do request                                    digest="sha256:3941d3b032a8168d53508410a67baad120a563df67a7959565a30a1cb2114731" mediatype=application/vnd.oci.image.config.v1+json request.header.accept="application/vnd.oci.image.config.v1+json, */*" request.header.user-agent=containerd/1.7.0-beta.3+unknown request.method=HEAD size=2297 url="https://self-hosted-proget.dropthought.com/v2/dt-onprem/ubuntu/blobs/sha256:3941d3b032a8168d53508410a67baad120a563df67a7959565a30a1cb2114731"
DEBU[0000] fetch response received                       digest="sha256:3941d3b032a8168d53508410a67baad120a563df67a7959565a30a1cb2114731" mediatype=application/vnd.oci.image.config.v1+json response.header.cache-control=private response.header.connection=keep-alive response.header.content-length=208 response.header.content-type=application/json response.header.date="Sun, 19 Mar 2023 17:43:03 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.vary="Accept-Encoding, Authorization" response.header.www-authenticate="Bearer realm=\"https://self-hosted-proget.dropthought.com/v2/_auth\",service=\"self-hosted-proget.dropthought.com\",scope=\"repository:dt-onprem/ubuntu:pull\"" response.header.x-proget-version=22.0.19.10 response.status="401 Unauthorized" size=2297 url="https://self-hosted-proget.dropthought.com/v2/dt-onprem/ubuntu/blobs/sha256:3941d3b032a8168d53508410a67baad120a563df67a7959565a30a1cb2114731"
DEBU[0000] Unauthorized                                  digest="sha256:3941d3b032a8168d53508410a67baad120a563df67a7959565a30a1cb2114731" header="Bearer realm=\"https://self-hosted-proget.dropthought.com/v2/_auth\",service=\"self-hosted-proget.dropthought.com\",scope=\"repository:dt-onprem/ubuntu:pull\"" mediatype=application/vnd.oci.image.config.v1+json size=2297
DEBU[0000] do request                                    digest="sha256:3941d3b032a8168d53508410a67baad120a563df67a7959565a30a1cb2114731" mediatype=application/vnd.oci.image.config.v1+json request.header.accept="application/vnd.oci.image.config.v1+json, */*" request.header.user-agent=containerd/1.7.0-beta.3+unknown request.method=HEAD size=2297 url="https://self-hosted-proget.dropthought.com/v2/dt-onprem/ubuntu/blobs/sha256:3941d3b032a8168d53508410a67baad120a563df67a7959565a30a1cb2114731"
DEBU[0000] fetch response received                       digest="sha256:0c5227665c11379f79e9da3d3e4f1724f9316b87d259ac0131628ca1b923a392" mediatype=application/vnd.oci.image.layer.v1.tar+gzip response.header.cache-control=private response.header.connection=keep-alive response.header.content-length=208 response.header.content-type=application/json response.header.date="Sun, 19 Mar 2023 17:43:03 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.vary="Accept-Encoding, Authorization" response.header.www-authenticate="Bearer realm=\"https://self-hosted-proget.dropthought.com/v2/_auth\",service=\"self-hosted-proget.dropthought.com\",scope=\"repository:dt-onprem/ubuntu:pull\"" response.header.x-proget-version=22.0.19.10 response.status="401 Unauthorized" size=25688211 url="https://self-hosted-proget.dropthought.com/v2/dt-onprem/ubuntu/blobs/sha256:0c5227665c11379f79e9da3d3e4f1724f9316b87d259ac0131628ca1b923a392"
DEBU[0000] Unauthorized                                  digest="sha256:0c5227665c11379f79e9da3d3e4f1724f9316b87d259ac0131628ca1b923a392" header="Bearer realm=\"https://self-hosted-proget.dropthought.com/v2/_auth\",service=\"self-hosted-proget.dropthought.com\",scope=\"repository:dt-onprem/ubuntu:pull\"" mediatype=application/vnd.oci.image.layer.v1.tar+gzip size=25688211
DEBU[0000] do request                                    digest="sha256:0c5227665c11379f79e9da3d3e4f1724f9316b87d259ac0131628ca1b923a392" mediatype=application/vnd.oci.image.layer.v1.tar+gzip request.header.accept="application/vnd.oci.image.layer.v1.tar+gzip, */*" request.header.user-agent=containerd/1.7.0-beta.3+unknown request.method=HEAD size=25688211 url="https://self-hosted-proget.dropthought.com/v2/dt-onprem/ubuntu/blobs/sha256:0c5227665c11379f79e9da3d3e4f1724f9316b87d259ac0131628ca1b923a392"
DEBU[0000] fetch response received                       digest="sha256:3941d3b032a8168d53508410a67baad120a563df67a7959565a30a1cb2114731" mediatype=application/vnd.oci.image.config.v1+json response.header.accept-ranges=bytes response.header.connection=keep-alive response.header.content-length=61583 response.header.content-type=text/html response.header.date="Sun, 19 Mar 2023 17:43:03 GMT" response.header.etag="\"5f02dd51-f08f\"" response.header.last-modified="Mon, 06 Jul 2020 08:14:09 GMT" response.status="200 OK" size=2297 url="https://self-hosted-proget.dropthought.com/v2/dt-onprem/ubuntu/blobs/sha256:3941d3b032a8168d53508410a67baad120a563df67a7959565a30a1cb2114731"
DEBU[0000] fetch response received                       digest="sha256:0c5227665c11379f79e9da3d3e4f1724f9316b87d259ac0131628ca1b923a392" mediatype=application/vnd.oci.image.layer.v1.tar+gzip response.header.accept-ranges=bytes response.header.connection=keep-alive response.header.content-length=61583 response.header.content-type=text/html response.header.date="Sun, 19 Mar 2023 17:43:03 GMT" response.header.etag="\"5f02dd51-f08f\"" response.header.last-modified="Mon, 06 Jul 2020 08:14:09 GMT" response.status="200 OK" size=25688211 url="https://self-hosted-proget.dropthought.com/v2/dt-onprem/ubuntu/blobs/sha256:0c5227665c11379f79e9da3d3e4f1724f9316b87d259ac0131628ca1b923a392"
DEBU[0000] push                                          digest="sha256:0779371f96205678dbcaa3ef499be2e5f262c8b09aadc11754bf3daf9f35e03e" mediatype=application/vnd.oci.image.manifest.v1+json size=424
DEBU[0000] checking and pushing to                       digest="sha256:0779371f96205678dbcaa3ef499be2e5f262c8b09aadc11754bf3daf9f35e03e" mediatype=application/vnd.oci.image.manifest.v1+json size=424 url="https://self-hosted-proget.dropthought.com/v2/dt-onprem/ubuntu/manifests/sha256:0779371f96205678dbcaa3ef499be2e5f262c8b09aadc11754bf3daf9f35e03e"
DEBU[0000] do request                                    digest="sha256:0779371f96205678dbcaa3ef499be2e5f262c8b09aadc11754bf3daf9f35e03e" mediatype=application/vnd.oci.image.manifest.v1+json request.header.accept="application/vnd.oci.image.manifest.v1+json, */*" request.header.user-agent=containerd/1.7.0-beta.3+unknown request.method=HEAD size=424 url="https://self-hosted-proget.dropthought.com/v2/dt-onprem/ubuntu/manifests/sha256:0779371f96205678dbcaa3ef499be2e5f262c8b09aadc11754bf3daf9f35e03e"
index-sha256:cfa36a700a8c12f0f16a243188e0731dd81d875fc266eb69b98de669cede4a4d:    waiting        |--------------------------------------|
manifest-sha256:0779371f96205678dbcaa3ef499be2e5f262c8b09aadc11754bf3daf9f35e03e: waiting        |--------------------------------------|
config-sha256:3941d3b032a8168d53508410a67baad120a563df67a7959565a30a1cb2114731:   waiting        |--------------------------------------|
elapsed: 0.1 s                                                                    total:   0.0 B (0.0 B/s)
DEBU[0000] fetch response received                       digest="sha256:0779371f96205678dbcaa3ef499be2e5f262c8b09aadc11754bf3daf9f35e03e" mediatype=application/vnd.oci.image.manifest.v1+json response.header.accept-ranges=bytes response.header.connection=keep-alive response.header.content-length=61583 response.header.content-type=text/html response.header.date="Sun, 19 Mar 2023 17:43:03 GMT" response.header.etag="\"5f02dd51-f08f\"" response.header.last-modified="Mon, 06 Jul 2020 08:14:09 GMT" response.status="200 OK" size=424 url="https://self-hosted-proget.dropthought.com/v2/dt-onprem/ubuntu/manifests/sha256:0779371f96205678dbcaa3ef499be2e5f262c8b09aadc11754bf3daf9f35e03e"
DEBU[0000] push                                          digest="sha256:cfa36a700a8c12f0f16a243188e0731dd81d875fc266eb69b98de669cede4a4d" mediatype=application/vnd.oci.image.index.v1+json size=289
DEBU[0000] checking and pushing to                       digest="sha256:cfa36a700a8c12f0f16a243188e0731dd81d875fc266eb69b98de669cede4a4d" mediatype=application/vnd.oci.image.index.v1+json size=289 url="https://self-hosted-proget.dropthought.com/v2/dt-onprem/ubuntu/manifests/bionic"
DEBU[0000] do request                                    digest="sha256:cfa36a700a8c12f0f16a243188e0731dd81d875fc266eb69b98de669cede4a4d" mediatype=application/vnd.oci.image.index.v1+json request.header.accept="application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.7.0-beta.3+unknown request.method=HEAD size=289 url="https://self-hosted-proget.dropthought.com/v2/dt-onprem/ubuntu/manifests/bionic"
DEBU[0000] fetch response received                       digest="sha256:cfa36a700a8c12f0f16a243188e0731dd81d875fc266eb69b98de669cede4a4d" mediatype=application/vnd.oci.image.index.v1+json response.header.accept-ranges=bytes response.header.connection=keeindex-sha256:cfa36a700a8c12f0f16a243188e0731dd81d875fc266eb69b98de669cede4a4d:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:0779371f96205678dbcaa3ef499be2e5f262c8b09aadc11754bf3daf9f35e03e: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:3941d3b032a8168d53508410a67baad120a563df67a7959565a30a1cb2114731:   done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 0.4 s                                                                    total:  2.9 Ki (7.3 KiB/s)
index-sha256:cfa36a700a8c12f0f16a243188e0731dd81d875fc266eb69b98de669cede4a4d:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:0779371f96205678dbcaa3ef499be2e5f262c8b09aadc11754bf3daf9f35e03e: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:3941d3b032a8168d53508410a67baad120a563df67a7959565a30a1cb2114731:   done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 0.6 s                                                                    total:  2.9 Ki (4.9 KiB/s)

atripp

Hello,

ProGet will issue a 401 (Unauthorized) when the request has not been authenticated or if the authenticated credentials are not correct.

We don't have instructions or troubleshooting steps for nerdctl, but in the docker client you will use a login first., before a push.

Here's some more info:
https://docs.inedo.com/docs/proget-docker-private-registries#creating-and-using-a-docker-registries-in-proget

Cheers,
Alana

mhdos_4222

Thank you. Nerdctl is a CLI replacement for docker which is compatible with docker CLI as well.

As I mentioned previously, the login succeeded before I attempted to push my image and it still resulted in the error.

Strangely, I face the same issue while attempting with docker-cli as well. The docker-cli used for this is 23.0.1, build a5ee5b1.

atripp

Hi @mhdos_4222 ,

Sorry, I must be confused.

Is the image actually pushed? It's not very clear from the logs, but it looks like it's working after you login? You wrote:

The push command was executed after successfully logging into our self hosted Proget.

A 401 basically just means the credentials are incorrect or not sent, and I really can't tell from the logs what's happening or not happening.

If things used to work before, do you know what was changed ? It's just hard to guess from the information we have here.

mhdos_4222

No, the image is not getting pushed successfully. For example, when I push an Ubuntu:bionic image (tagged with the self hosted registry), I see the following reported under the "Container Configuration" on the page that lists the tags for Ubuntu.

A container configuration file for this image could not loaded, which means configuration metadata is not be available. This may be expected, as these files are optional and, depending on how the image was created, may not be present.

The following is the Raw manifest Json

{"schemaVersion":2,"mediaType":"application/vnd.oci.image.index.v1+json","manifests":[{"mediaType":"application/vnd.oci.image.manifest.v1+json","digest":"sha256:0779371f96205678dbcaa3ef499be2e5f262c8b09aadc11754bf3daf9f35e03e","size":424,"platform":{"architecture":"amd64","os":"linux"}}]}

When I click on Raw Configuration Json, it's listed as unavailable.

mhdos_4222

Kindly consider this closed.

I deleted my nginx ingress controller completely and redeployed it from scratch (including the ingress) and voila! I was able to push and pull images again.

I suspect my ingress controller might have got wonky, but its logs didn't show anything out of the norm.