Hello @atripp

Thanks for sharing the code segment. I believe that the issue could be somewhere else and not related to Proget at this point.

I attempted to push to Amazon ECR using the same nerdcrl and found that I'm able to push it just fine. This leads me to believe the problem is not with nerdctl or docker cli for that matter.

I came across this thread which I feel does not serve as a proper documentation (but relatable though).

Bottomline, I wouldn't want to break anything that is working fine.
I'm going to deploy Proget on AWS Lightsail and compare with what's going on the k8s deployment.

Btw, I upgraded my Proget to 22.0.26 and the result was the same.

Hello @atripp ,

Thanks. I dug a little more and found this as part of the debug mode from nerdctl

DEBU[0000] fetch response received                       digest="sha256:f1b5933fe4b5f49bbe8258745cf396afe07e625bdab3168e364daf7c956b6b81" mediatype=application/vnd.docker.image.rootfs.diff.tar response.header.cache-control=private response.header.connection=keep-alive response.header.content-length=235 response.header.content-type=application/json response.header.date="Mon, 27 Mar 2023 16:54:51 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.vary="Accept-Encoding, Authorization" response.header.www-authenticate="Bearer realm=\"https://test-proget.xxx.com/v2/_auth\",service=\"test-proget.xxx.com\",scope=\"repository:test-feed/platform/service/eml-service:pull\"" response.header.x-proget-version=22.0.19.10 response.status="401 Unauthorized" size=5796352 url="https://test-proget.xxx.com/v2/test-feed/platform/service/eml-service/blobs/sha256:f1b5933fe4b5f49bbe8258745cf396afe07e625bdab3168e364daf7c956b6b81"

I can see that the scope is set to pull instead of pull,push

We have deployed a free edition of Proget 2022.19 (Build 10) on our self hosted K8s. The K8s cluster runs on ec2 instances hosted in AWS and is exposed via an classic ELB on AWS to an https endpoint. The WebBase.URL is also set to the URL of the self-hosted Proget.

Until last week, we were able to push images and pull from our self hosted Proget registry without any issues. However, since last Friday, we are unable to push or pull images from the registry. We attempted with a fresh installation after clearing the persistent volume behind MSSQL and Proget, but the results were the same.

The K8s cluster is of version 1.21.4 and uses containerd 1.7.0 as the container runtime. We also user nerdctl instead of docker-cli to tag and push images as required.

I already opened a topic here which was solved when I removed and redeployed my ingress-controller (kubernetes-nginx). Unfortunately, it doesn't work this time.

Here are some images from the self-hosted proget after I push the image using nerdctl that mention about an issue with the configuration file associated with the image.

I also attempted to push the same image using docker-cli v23.0.1 from another machine. Strangely, the first 3 layers are reported as missing on the GUI If I do so. If I use nerdctl instead of docker-cli, the last layer is reported as missing.

Ultimately, I can't pull an image from Proget after I push it - though the push command itself doesn't report any errors (either from docker-cli or from nerdctl).

Can someone shed some light on what I'm missing?

Kindly consider this closed.

I deleted my nginx ingress controller completely and redeployed it from scratch (including the ingress) and voila! I was able to push and pull images again.

I suspect my ingress controller might have got wonky, but its logs didn't show anything out of the norm.

No, the image is not getting pushed successfully. For example, when I push an Ubuntu:bionic image (tagged with the self hosted registry), I see the following reported under the "Container Configuration" on the page that lists the tags for Ubuntu.

A container configuration file for this image could not loaded, which means configuration metadata is not be available. This may be expected, as these files are optional and, depending on how the image was created, may not be present.

The following is the Raw manifest Json

{"schemaVersion":2,"mediaType":"application/vnd.oci.image.index.v1+json","manifests":[{"mediaType":"application/vnd.oci.image.manifest.v1+json","digest":"sha256:0779371f96205678dbcaa3ef499be2e5f262c8b09aadc11754bf3daf9f35e03e","size":424,"platform":{"architecture":"amd64","os":"linux"}}]}

When I click on Raw Configuration Json, it's listed as unavailable.

Thank you. Nerdctl is a CLI replacement for docker which is compatible with docker CLI as well.

As I mentioned previously, the login succeeded before I attempted to push my image and it still resulted in the error.

Strangely, I face the same issue while attempting with docker-cli as well. The docker-cli used for this is 23.0.1, build a5ee5b1.

We have deployed a free edition of Proget 2022.19 (Build 10) on our self hosted K8s. The K8s cluster runs on ec2 instances hosted in AWS and is exposed via an classic ELB on AWS to an https endpoint. The WebBase.URL is also set to the URL of the self-hosted Proget.

Until last week, we were able to push images and pull from our self hosted Proget registry without any issues. However, since last Friday, we are unable to push or pull images from the registry. We attempted with a fresh installation after clearing the persistent volume behind MSSQL and Proget, but the results were the same.

The K8s cluster is of version 1.21.4 and uses containerd 1.7.0 as the container runtime.

Given below is the debug output of the push command. I can see several Unauthorized entries, but unsure why they are there. The push command was executed after successfully logging into our self hosted Proget.

Any ideas on what is happening?

ubuntu@ip-10-0-2-68:~/proget$ sudo nerdctl --debug-full image push self-hosted-proget.dropthought.com/dt-onprem/ubuntu:bionic
DEBU[0000] converted                                     new="<nil>" old="{application/vnd.oci.image.layer.v1.tar+gzip sha256:0c5227665c11379f79e9da3d3e4f1724f9316b87d259ac0131628ca1b923a392 25688211 [] map[] [] <nil> }"
DEBU[0000] converted                                     new="<nil>" old="{application/vnd.oci.image.config.v1+json sha256:3941d3b032a8168d53508410a67baad120a563df67a7959565a30a1cb2114731 2297 [] map[] [] <nil> }"
DEBU[0000] converted                                     new="<nil>" old="{application/vnd.oci.image.manifest.v1+json sha256:0779371f96205678dbcaa3ef499be2e5f262c8b09aadc11754bf3daf9f35e03e 424 [] map[] [] 0xc00002be60 }"
DEBU[0000] converted                                     new="&{application/vnd.oci.image.index.v1+json sha256:cfa36a700a8c12f0f16a243188e0731dd81d875fc266eb69b98de669cede4a4d 289 [] map[] [] <nil> }" old="{application/vnd.oci.image.index.v1+json sha256:8aa9c2798215f99544d1ce7439ea9c3a6dfd82de607da1cec3a8a2fae005931b 1334 [] map[] [] <nil> }"
INFO[0000] pushing as a reduced-platform image (application/vnd.oci.image.index.v1+json, sha256:cfa36a700a8c12f0f16a243188e0731dd81d875fc266eb69b98de669cede4a4d)
DEBU[0000] Ignoring hosts dir "/etc/containerd/certs.d"  error="stat /etc/containerd/certs.d: no such file or directory"
DEBU[0000] Ignoring hosts dir "/etc/docker/certs.d"      error="stat /etc/docker/certs.d: no such file or directory"
DEBU[0000] pushing                                       digest="sha256:cfa36a700a8c12f0f16a243188e0731dd81d875fc266eb69b98de669cede4a4d" image="self-hosted-proget.dropthought.com/dt-onprem/ubuntu:bionic"
DEBU[0000] push                                          digest="sha256:0c5227665c11379f79e9da3d3e4f1724f9316b87d259ac0131628ca1b923a392" mediatype=application/vnd.oci.image.layer.v1.tar+gzip size=25688211
DEBU[0000] checking and pushing to                       digest="sha256:0c5227665c11379f79e9da3d3e4f1724f9316b87d259ac0131628ca1b923a392" mediatype=application/vnd.oci.image.layer.v1.tar+gzip size=25688211 url="https://self-hosted-proget.dropthought.com/v2/dt-onprem/ubuntu/blobs/sha256:0c5227665c11379f79e9da3d3e4f1724f9316b87d259ac0131628ca1b923a392"
DEBU[0000] do request                                    digest="sha256:0c5227665c11379f79e9da3d3e4f1724f9316b87d259ac0131628ca1b923a392" mediatype=application/vnd.oci.image.layer.v1.tar+gzip request.header.accept="application/vnd.oci.image.layer.v1.tar+gzip, */*" request.header.user-agent=containerd/1.7.0-beta.3+unknown request.method=HEAD size=25688211 url="https://self-hosted-proget.dropthought.com/v2/dt-onprem/ubuntu/blobs/sha256:0c5227665c11379f79e9da3d3e4f1724f9316b87d259ac0131628ca1b923a392"
DEBU[0000] push                                          digest="sha256:3941d3b032a8168d53508410a67baad120a563df67a7959565a30a1cb2114731" mediatype=application/vnd.oci.image.config.v1+json size=2297
DEBU[0000] checking and pushing to                       digest="sha256:3941d3b032a8168d53508410a67baad120a563df67a7959565a30a1cb2114731" mediatype=application/vnd.oci.image.config.v1+json size=2297 url="https://self-hosted-proget.dropthought.com/v2/dt-onprem/ubuntu/blobs/sha256:3941d3b032a8168d53508410a67baad120a563df67a7959565a30a1cb2114731"
DEBU[0000] do request                                    digest="sha256:3941d3b032a8168d53508410a67baad120a563df67a7959565a30a1cb2114731" mediatype=application/vnd.oci.image.config.v1+json request.header.accept="application/vnd.oci.image.config.v1+json, */*" request.header.user-agent=containerd/1.7.0-beta.3+unknown request.method=HEAD size=2297 url="https://self-hosted-proget.dropthought.com/v2/dt-onprem/ubuntu/blobs/sha256:3941d3b032a8168d53508410a67baad120a563df67a7959565a30a1cb2114731"
DEBU[0000] fetch response received                       digest="sha256:3941d3b032a8168d53508410a67baad120a563df67a7959565a30a1cb2114731" mediatype=application/vnd.oci.image.config.v1+json response.header.cache-control=private response.header.connection=keep-alive response.header.content-length=208 response.header.content-type=application/json response.header.date="Sun, 19 Mar 2023 17:43:03 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.vary="Accept-Encoding, Authorization" response.header.www-authenticate="Bearer realm=\"https://self-hosted-proget.dropthought.com/v2/_auth\",service=\"self-hosted-proget.dropthought.com\",scope=\"repository:dt-onprem/ubuntu:pull\"" response.header.x-proget-version=22.0.19.10 response.status="401 Unauthorized" size=2297 url="https://self-hosted-proget.dropthought.com/v2/dt-onprem/ubuntu/blobs/sha256:3941d3b032a8168d53508410a67baad120a563df67a7959565a30a1cb2114731"
DEBU[0000] Unauthorized                                  digest="sha256:3941d3b032a8168d53508410a67baad120a563df67a7959565a30a1cb2114731" header="Bearer realm=\"https://self-hosted-proget.dropthought.com/v2/_auth\",service=\"self-hosted-proget.dropthought.com\",scope=\"repository:dt-onprem/ubuntu:pull\"" mediatype=application/vnd.oci.image.config.v1+json size=2297
DEBU[0000] do request                                    digest="sha256:3941d3b032a8168d53508410a67baad120a563df67a7959565a30a1cb2114731" mediatype=application/vnd.oci.image.config.v1+json request.header.accept="application/vnd.oci.image.config.v1+json, */*" request.header.user-agent=containerd/1.7.0-beta.3+unknown request.method=HEAD size=2297 url="https://self-hosted-proget.dropthought.com/v2/dt-onprem/ubuntu/blobs/sha256:3941d3b032a8168d53508410a67baad120a563df67a7959565a30a1cb2114731"
DEBU[0000] fetch response received                       digest="sha256:0c5227665c11379f79e9da3d3e4f1724f9316b87d259ac0131628ca1b923a392" mediatype=application/vnd.oci.image.layer.v1.tar+gzip response.header.cache-control=private response.header.connection=keep-alive response.header.content-length=208 response.header.content-type=application/json response.header.date="Sun, 19 Mar 2023 17:43:03 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.vary="Accept-Encoding, Authorization" response.header.www-authenticate="Bearer realm=\"https://self-hosted-proget.dropthought.com/v2/_auth\",service=\"self-hosted-proget.dropthought.com\",scope=\"repository:dt-onprem/ubuntu:pull\"" response.header.x-proget-version=22.0.19.10 response.status="401 Unauthorized" size=25688211 url="https://self-hosted-proget.dropthought.com/v2/dt-onprem/ubuntu/blobs/sha256:0c5227665c11379f79e9da3d3e4f1724f9316b87d259ac0131628ca1b923a392"
DEBU[0000] Unauthorized                                  digest="sha256:0c5227665c11379f79e9da3d3e4f1724f9316b87d259ac0131628ca1b923a392" header="Bearer realm=\"https://self-hosted-proget.dropthought.com/v2/_auth\",service=\"self-hosted-proget.dropthought.com\",scope=\"repository:dt-onprem/ubuntu:pull\"" mediatype=application/vnd.oci.image.layer.v1.tar+gzip size=25688211
DEBU[0000] do request                                    digest="sha256:0c5227665c11379f79e9da3d3e4f1724f9316b87d259ac0131628ca1b923a392" mediatype=application/vnd.oci.image.layer.v1.tar+gzip request.header.accept="application/vnd.oci.image.layer.v1.tar+gzip, */*" request.header.user-agent=containerd/1.7.0-beta.3+unknown request.method=HEAD size=25688211 url="https://self-hosted-proget.dropthought.com/v2/dt-onprem/ubuntu/blobs/sha256:0c5227665c11379f79e9da3d3e4f1724f9316b87d259ac0131628ca1b923a392"
DEBU[0000] fetch response received                       digest="sha256:3941d3b032a8168d53508410a67baad120a563df67a7959565a30a1cb2114731" mediatype=application/vnd.oci.image.config.v1+json response.header.accept-ranges=bytes response.header.connection=keep-alive response.header.content-length=61583 response.header.content-type=text/html response.header.date="Sun, 19 Mar 2023 17:43:03 GMT" response.header.etag="\"5f02dd51-f08f\"" response.header.last-modified="Mon, 06 Jul 2020 08:14:09 GMT" response.status="200 OK" size=2297 url="https://self-hosted-proget.dropthought.com/v2/dt-onprem/ubuntu/blobs/sha256:3941d3b032a8168d53508410a67baad120a563df67a7959565a30a1cb2114731"
DEBU[0000] fetch response received                       digest="sha256:0c5227665c11379f79e9da3d3e4f1724f9316b87d259ac0131628ca1b923a392" mediatype=application/vnd.oci.image.layer.v1.tar+gzip response.header.accept-ranges=bytes response.header.connection=keep-alive response.header.content-length=61583 response.header.content-type=text/html response.header.date="Sun, 19 Mar 2023 17:43:03 GMT" response.header.etag="\"5f02dd51-f08f\"" response.header.last-modified="Mon, 06 Jul 2020 08:14:09 GMT" response.status="200 OK" size=25688211 url="https://self-hosted-proget.dropthought.com/v2/dt-onprem/ubuntu/blobs/sha256:0c5227665c11379f79e9da3d3e4f1724f9316b87d259ac0131628ca1b923a392"
DEBU[0000] push                                          digest="sha256:0779371f96205678dbcaa3ef499be2e5f262c8b09aadc11754bf3daf9f35e03e" mediatype=application/vnd.oci.image.manifest.v1+json size=424
DEBU[0000] checking and pushing to                       digest="sha256:0779371f96205678dbcaa3ef499be2e5f262c8b09aadc11754bf3daf9f35e03e" mediatype=application/vnd.oci.image.manifest.v1+json size=424 url="https://self-hosted-proget.dropthought.com/v2/dt-onprem/ubuntu/manifests/sha256:0779371f96205678dbcaa3ef499be2e5f262c8b09aadc11754bf3daf9f35e03e"
DEBU[0000] do request                                    digest="sha256:0779371f96205678dbcaa3ef499be2e5f262c8b09aadc11754bf3daf9f35e03e" mediatype=application/vnd.oci.image.manifest.v1+json request.header.accept="application/vnd.oci.image.manifest.v1+json, */*" request.header.user-agent=containerd/1.7.0-beta.3+unknown request.method=HEAD size=424 url="https://self-hosted-proget.dropthought.com/v2/dt-onprem/ubuntu/manifests/sha256:0779371f96205678dbcaa3ef499be2e5f262c8b09aadc11754bf3daf9f35e03e"
index-sha256:cfa36a700a8c12f0f16a243188e0731dd81d875fc266eb69b98de669cede4a4d:    waiting        |--------------------------------------|
manifest-sha256:0779371f96205678dbcaa3ef499be2e5f262c8b09aadc11754bf3daf9f35e03e: waiting        |--------------------------------------|
config-sha256:3941d3b032a8168d53508410a67baad120a563df67a7959565a30a1cb2114731:   waiting        |--------------------------------------|
elapsed: 0.1 s                                                                    total:   0.0 B (0.0 B/s)
DEBU[0000] fetch response received                       digest="sha256:0779371f96205678dbcaa3ef499be2e5f262c8b09aadc11754bf3daf9f35e03e" mediatype=application/vnd.oci.image.manifest.v1+json response.header.accept-ranges=bytes response.header.connection=keep-alive response.header.content-length=61583 response.header.content-type=text/html response.header.date="Sun, 19 Mar 2023 17:43:03 GMT" response.header.etag="\"5f02dd51-f08f\"" response.header.last-modified="Mon, 06 Jul 2020 08:14:09 GMT" response.status="200 OK" size=424 url="https://self-hosted-proget.dropthought.com/v2/dt-onprem/ubuntu/manifests/sha256:0779371f96205678dbcaa3ef499be2e5f262c8b09aadc11754bf3daf9f35e03e"
DEBU[0000] push                                          digest="sha256:cfa36a700a8c12f0f16a243188e0731dd81d875fc266eb69b98de669cede4a4d" mediatype=application/vnd.oci.image.index.v1+json size=289
DEBU[0000] checking and pushing to                       digest="sha256:cfa36a700a8c12f0f16a243188e0731dd81d875fc266eb69b98de669cede4a4d" mediatype=application/vnd.oci.image.index.v1+json size=289 url="https://self-hosted-proget.dropthought.com/v2/dt-onprem/ubuntu/manifests/bionic"
DEBU[0000] do request                                    digest="sha256:cfa36a700a8c12f0f16a243188e0731dd81d875fc266eb69b98de669cede4a4d" mediatype=application/vnd.oci.image.index.v1+json request.header.accept="application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.7.0-beta.3+unknown request.method=HEAD size=289 url="https://self-hosted-proget.dropthought.com/v2/dt-onprem/ubuntu/manifests/bionic"
DEBU[0000] fetch response received                       digest="sha256:cfa36a700a8c12f0f16a243188e0731dd81d875fc266eb69b98de669cede4a4d" mediatype=application/vnd.oci.image.index.v1+json response.header.accept-ranges=bytes response.header.connection=keeindex-sha256:cfa36a700a8c12f0f16a243188e0731dd81d875fc266eb69b98de669cede4a4d:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:0779371f96205678dbcaa3ef499be2e5f262c8b09aadc11754bf3daf9f35e03e: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:3941d3b032a8168d53508410a67baad120a563df67a7959565a30a1cb2114731:   done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 0.4 s                                                                    total:  2.9 Ki (7.3 KiB/s)
index-sha256:cfa36a700a8c12f0f16a243188e0731dd81d875fc266eb69b98de669cede4a4d:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:0779371f96205678dbcaa3ef499be2e5f262c8b09aadc11754bf3daf9f35e03e: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:3941d3b032a8168d53508410a67baad120a563df67a7959565a30a1cb2114731:   done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 0.6 s                                                                    total:  2.9 Ki (4.9 KiB/s)