500 Internal Server Error when pushing docker image

[Stephen.Schaff]

I have been able to successfully push docker images for a while now. Most of the images I was pushing were all to the same feed's docker repository. Today I needed to push to a new docker repository in the same feed.

When I did that I got this error:

received unexpected HTTP status: 500 Internal Server Error

I went back to an existing repository and tried deleting a tag and repushing it. That worked fine.

Something is going wrong when I use a new repository name.

I went to the administration page in ProGet and looked in the Diagnostic Center and there are no logs during the time that the error occurred.

How can I get additional details on this error so I can go about diagnosing it?

---

Here is what my attempt to push the docker image looked like:

[builderuser@SWDGNUBUILD01N ~]$ docker login proget.mydomain.net
Username: proget-admin
Password:
WARNING! Your password will be stored unencrypted in /home/builderuser/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
[builderuser@SWDGNUBUILD01N ~]$ docker push proget.mydomain.net/application-prerelease/runtime/my-application-service:1.9.0-build.147
The push refers to repository [proget.mydomain.net/application-prerelease/runtime/my-application-service]
d1969e3e3bf7: Layer already exists
d79cf2683c2b: Layer already exists
fdce96d81153: Layer already exists
379b6cbd8951: Layer already exists
5c65967e47cc: Layer already exists
b49c0ab385b6: Layer already exists
bd13a0d30d03: Layer already exists
49a130b8b29d: Layer already exists
68f8345afd82: Layer already exists
4466146b6e2c: Layer already exists
8b9bd48d3c62: Layer already exists
fe5e09df9bed: Layer already exists
86248bc7c05e: Layer already exists
093496e1b286: Layer already exists
139aa1a59c8a: Layer already exists
f3bc884425c5: Layer already exists
6db0849b3e99: Layer already exists
800afb4d883b: Layer already exists
cb42413394c4: Layer already exists
received unexpected HTTP status: 500 Internal Server Error

Also, this was run from a Linux machine (in case that matters).

[Stephen.Schaff]

I dug into this more and eventually found this error:

Image_Digest must be unique across the containing feed.

It would appear that the fact that I had this same image in another feed made it fail when I tried to put it into a new feed with a new image name and tag.

When I went to my source code and added a whitespace to a line and rebuilt the docker image it worked fine.

I would like to suggest that this could be better. At the least, it would be nice if this error was more discoverable. I had to turn IIS logging (using this guide). Once I did that, the logs said that it was a status code 500 with an ErrorCode of "The operation completed successfully. (0x0)". (Not very helpful :)

Eventually I was able to dig through the lower level logs till I found this:

{"errors":[{"code":"UNKNOWN","message":"50000`16`1`TR__DockerImages__ValidateUniqueDigest`19`Image_Digest must be unique across the containing feed.","detail":[]}]}

Ideally this would show up in the Administration page's Diagnostic Center. Failing that, it would be nice if it was in the event log for the host machine.

I also have to wonder why this limitation exists. I thought docker could have the same image in different repositories any number of times.

[viceice]

Seeing 500 on https://proget.inedo.com/containers/registry?feedId=22.

Maybe that is related?

[viceice]

This is the error i get:

"options": {
      "headers": {
        "user-agent": "Renovate Bot (GitHub App 2740)",
        "accept-encoding": "gzip, deflate",
        "authorization": "***********",
        "accept": "application/json"
      },
      "url": "https://proget.inedo.com/v2/productimages/inedo/proget/tags/list?n=10000",
      "username": "",
      "password": "",
      "method": "GET",
      "http2": false
    },
    "response": {
      "statusCode": 500,
      "statusMessage": "Internal Server Error",
      "body": null,
      "headers": {
        "cache-control": "public, max-age=300",
        "content-type": "application/json; charset=utf-8",
        "content-encoding": "gzip",
        "vary": "Authorization",
        "server": "Microsoft-IIS/10.0",
        "x-proget-version": "5.3.27.18",
        "docker-distribution-api-version": "registry/2.0",
        "x-aspnet-version": "4.0.30319",
        "x-powered-by": "ASP.NET",
        "date": "Wed, 21 Apr 2021 05:57:33 GMT",
        "connection": "close",
        "content-length": "84"
      },
      "httpVersion": "1.1"
    }
  }

[atripp]

Hi @Stephen-Schaff,

An error like that should be better reported by Docker (that XML is their format), but it also should have appeared in the diagnostic center, since it's an unexpected server error. It's likely suppressed by mistake, for expected errors like "tag not found" and the like.

As for why the error occurred, it's hard to say -- but definitely a bug. Do both of your feeds have the same storage type? Under Manage Feed > Storage?

There is some complexity with "common blog storage", so it's possibly related to that. If you can share info, we can try to repro and then fix.

@viceice thanks for reporting that. It's unrelated. The error message I see (in Admin > Diagnostic Center) is implying that the problem is related to our database structure.... perhaps from installing a pre-release version? We use fairly unstable versions for our internal production environments :)

[Stephen.Schaff]

Do both of your feeds have the same storage type? Under Manage Feed > Storage?

I misspoke when I said I was using two feeds. It was all in the same feed, I was just trying to make a new docker repository in the same feed. (Basically, I wanted to change the name of my container image by doing a docker push with it having a different name. But it was all to the same ProGet feed.

Here are the storage properties of that feed:

[atripp]

Thanks @Stephen-Schaff! FYI we are currently investigating this, and I hope to hear back soon. I'm starting think that maybe this is the issue, but I'd like to confirm and fix it first...

[rhessinger]

Hi @Stephen-Schaff,

I was finally able to recreate this error and I believe I fixed it in PG-1948. This is set to be released on Friday in ProGet 5.3.28. If you would like to apply the fix now, I have attached a SQL script to PG-1948 and you can run it against your ProGet database. If you run this script, it will not affect future upgrades of ProGet.

Thanks,
Rich