Issue with container configuration file on image upload

mhdos_4222

We have deployed a free edition of Proget 2022.19 (Build 10) on our self hosted K8s. The K8s cluster runs on ec2 instances hosted in AWS and is exposed via an classic ELB on AWS to an https endpoint. The WebBase.URL is also set to the URL of the self-hosted Proget.

Until last week, we were able to push images and pull from our self hosted Proget registry without any issues. However, since last Friday, we are unable to push or pull images from the registry. We attempted with a fresh installation after clearing the persistent volume behind MSSQL and Proget, but the results were the same.

The K8s cluster is of version 1.21.4 and uses containerd 1.7.0 as the container runtime. We also user nerdctl instead of docker-cli to tag and push images as required.

I already opened a topic here which was solved when I removed and redeployed my ingress-controller (kubernetes-nginx). Unfortunately, it doesn't work this time.

Here are some images from the self-hosted proget after I push the image using nerdctl that mention about an issue with the configuration file associated with the image.

I also attempted to push the same image using docker-cli v23.0.1 from another machine. Strangely, the first 3 layers are reported as missing on the GUI If I do so. If I use nerdctl instead of docker-cli, the last layer is reported as missing.

Ultimately, I can't pull an image from Proget after I push it - though the push command itself doesn't report any errors (either from docker-cli or from nerdctl).

Can someone shed some light on what I'm missing?

atripp

Hi @mhdos_4222 ,

I'm afraid I'm not sure how to help troubleshoot this very well; behind the scenes, the server (i.e. ProGet) side of things is fairly simple. Images are basically just manifest files and blobs, and ProGet will add those to the repository when receiving commands to a somewhat basic REST API.

What I would do is use a tool like Fiddler to capture the HTTP traffic between the client and server, and see if you can identify if there are any failed or missing requests. For example, maybe the client is never uploading 3cc66... for some reason. Or perhaps, the request is getting "eaten" but your ingress-controller for some reason.

I would also try to take your somewhat complicated configuration out of the equation, and just go with the most basic setup possible, like this: https://docs.inedo.com/docs/proget-how-to-install-on-aws-lightsail

You can then compare/contrast the HTTP traffic and find where there are issues.
Cheers,
Alana

mhdos_4222

Hello @atripp ,

Thanks. I dug a little more and found this as part of the debug mode from nerdctl

DEBU[0000] fetch response received                       digest="sha256:f1b5933fe4b5f49bbe8258745cf396afe07e625bdab3168e364daf7c956b6b81" mediatype=application/vnd.docker.image.rootfs.diff.tar response.header.cache-control=private response.header.connection=keep-alive response.header.content-length=235 response.header.content-type=application/json response.header.date="Mon, 27 Mar 2023 16:54:51 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.vary="Accept-Encoding, Authorization" response.header.www-authenticate="Bearer realm=\"https://test-proget.xxx.com/v2/_auth\",service=\"test-proget.xxx.com\",scope=\"repository:test-feed/platform/service/eml-service:pull\"" response.header.x-proget-version=22.0.19.10 response.status="401 Unauthorized" size=5796352 url="https://test-proget.xxx.com/v2/test-feed/platform/service/eml-service/blobs/sha256:f1b5933fe4b5f49bbe8258745cf396afe07e625bdab3168e364daf7c956b6b81"

I can see that the scope is set to pull instead of pull,push

atripp

Hi @mhdos_4222 ,

I'm not sure if that's related (and it was, why changing network would have any impact last time).

When the current user is not authorized to perform the required task in a Docker feed, this is code that ProGet runs:

string[] scopeParts = new[] { "repository", fullRepositoryName, ex.SecuredTask == (int)ProGetSecuredTask.Feeds_ViewFeed ? "pull" : "push" };
context.Response.AppendHeader("WWW-Authenticate", $@"Bearer realm=""{authUrl}"",service=""{context.Request.Url.Host}"",scope=""{string.Join(":", scopeParts)}""");
WriteError(context, new DockerException(401, "UNAUTHORIZED", ex.Message), feed, w =>
{
    w.WriteStartObject();
    w.WritePropertyName("Type");
    w.WriteValue(scopeParts[0]);
    w.WritePropertyName("Name");
    w.WriteValue(scopeParts[1]);
    w.WritePropertyName("Action");
    w.WriteValue(scopeParts[2]);
    w.WriteEndObject();
});

I do know that the Docker authentication stuff is very sensitive/complex, and I don't think anyone can answer why ProGet writes pull instead of pull,push. But whatever we're doing now works...

This code has basically been the same for about five years now, and we don't want to just change it because that will probably cause something to break.

Do you have any documentation/evidence/specs or anything that points to pull,push being a correct response?

Thanks,
Alana

mhdos_4222

Hello @atripp

Thanks for sharing the code segment. I believe that the issue could be somewhere else and not related to Proget at this point.

I attempted to push to Amazon ECR using the same nerdcrl and found that I'm able to push it just fine. This leads me to believe the problem is not with nerdctl or docker cli for that matter.

I came across this thread which I feel does not serve as a proper documentation (but relatable though).

Bottomline, I wouldn't want to break anything that is working fine.
I'm going to deploy Proget on AWS Lightsail and compare with what's going on the k8s deployment.

Btw, I upgraded my Proget to 22.0.26 and the result was the same.