Violation of PRIMARY KEY constraint 'PK__DockerBlobIndex'. Cannot insert duplicate key in object 'dbo.DockerImageLayers'.

[james.traxler_1560]

Proget gives the following error when I re-add an image that has multiple Docker image layers:

Violation of PRIMARY KEY constraint 'PK__DockerBlobIndex'. Cannot insert duplicate key in object 'dbo.DockerImageLayers'. The duplicate key value is (47, 1).\r\n

Any idea what is causing this?

[james.traxler_1560]

Note: This occurs when sending a request to

PUT /v2/<name>/manifests/<reference>

As per https://docs.docker.com/registry/spec/api/#pushing-an-image-manifest

[atripp]

Hi @james-traxler_1560 ,

We can help with this...

In general, the error makes sense if you're trying to re-add an image that's already there... but can you describe how you're doing that?

I.e. are you using the Docker client to push an image, or using the API directly? execute the PUT statement?

What is the version of ProGet that you're using?

Can you give a full stack trace (not sure if there's an error)?

Thanks,
Alana

[james.traxler_1560]

Hi @atripp

We are using ProGet Version 5.3.28 (Build 16), and calling the API directly.

Usage scenario: removing or renaming tags for a manifest. As there is no specific route for this use case, we are DELETE-ing the manifest through the v2 docker API and re-adding (with PUT) the new/remaining tags.

Basically, we are batching (so, executing in parallel) PUT requests to the URL with the format:

https://our.proget.com/v2/<feed>/<repository>/manifest/<tag>

e.g. with the following values:
• feed: qa-feed-1
• repository: library/nginx
• tag: in [latest, 1, 2, 3, test, test2]

we are concurrently executing 6 PUT requests:

PUT https://our.proget.com/v2/qa-feed-1/library/nginx/manifest/latest
PUT https://our.proget.com/v2/qa-feed-1/library/nginx/manifest/1
PUT https://our.proget.com/v2/qa-feed-1/library/nginx/manifest/2
PUT https://our.proget.com/v2/qa-feed-1/library/nginx/manifest/3
PUT https://our.proget.com/v2/qa-feed-1/library/nginx/manifest/test
PUT https://our.proget.com/v2/qa-feed-1/library/nginx/manifest/test2

In general, all of them return with a code 201, but one (random one) returns with a 500. Sometimes, all PUTs return with a code 201 (and that’s good as it is what we expect).

Response payload on HTTP 500:

{
   "errors":[
      {
         "code":"UNKNOWN",
         "message":"Violation of PRIMARY KEY constraint 'PK__DockerBlobIndex'. Cannot insert duplicate key in object 'dbo.DockerImageLayers'. The duplicate key value is (107, 1).\r\nThe statement has been terminated.",
         "detail":[]
      }
   ]
}

Our test images are the official nginx and hello-world.

I can forward a video we've taken of this in action - let me know an email address to send to (as I can't attach the mp4 here).

And here is a screenshot from the video, recapping the DELETE/PUT requests that occurred during the video:

Screenshot caption (use waterfall for request groups):

• DELETE + concurrent PUTs = “retag” action or “remove a single tag” action
• Single PUT = create new tag for manifest (top-right form of the page)

Also, sometimes only PUT-ing a single manifest (same URL as above) after a
DELETE on https://our.proget.com/v2/qa-feed-1/library/nginx/manifest/sha:xxxxx
results in the PUT returning HTTP 500.
This occurs more often with larger images, but not every time.

As a side note (unrelated to this error), GET requests on https://our.proget.com/v2/qa-feed-1/library/nginx/manifest/latest with header Accept: application/vnd.docker.distribution.manifest.v1+json return a payload of type application/vnd.docker.distribution.manifest.v2+json which doesn’t contain the data we are expecting (e.g. we can’t explore the layers of the manifest through the requested version of this manifest).

See screenshot:

Note: Regarding requested URL: /api/registries/1 part of the URL is only a backend proxy, passing the request as-is to the Proget API – basically transforming http://localhost:9000/api/registries/1/v2/… to https://our.proget.com/v2/…

[atripp]

Hi @james-traxler_1560,

Thanks for all of the info!

We are using ProGet Version 5.3.28 (Build 16), and calling the API directly.

Ah, that explains it. We really only test this with the Docker client, which wouldn't trigger this issue.

Usage scenario: removing or renaming tags for a manifest. As there is no specific route for this use case, we are DELETE-ing the manifest through the v2 docker API and re-adding (with PUT) the new/remaining tags.

Is this a Docker API limitation? We were thinking of adding a API for tag management for our own CI/CD platform (BuildMaster), but just worked-around it, and no one ever asked.

we are concurrently executing 6 PUT requests:

This is probably where the issue is. Looking at the database code, there is an opportunity for this to happen between a DELETE and INSERT statement.

It's pretty easy to fix, can you give it a shot? If it works in your test, then I'll commit the change. But note this will be for v6.0.15, so unless you upgrade to that or later, then you won't have the patched code.

https://inedo.myjetbrains.com/youtrack/issue/PG-2140

If you download the file attached to the above link, you can run it against your SQL Server database.

Thanks,
Alana

[james.traxler_1560]

Hi @atripp

We loaded your SQL Server script and tested and it did resolve the DELETE + re-INSERT issue!
So, if you do include that in your next patch release, that'd work - thank you.

Regarding this:

Is this a Docker API limitation? We were thinking of adding a API for tag management for our own CI/CD platform (BuildMaster), but just worked-around it, and no one ever asked.

Yes, it is a Docker API limitation.
It might be a good idea to have tag management API endpoints but it's not something we need specifically, so I wouldn't worry.

Thank you
James

[atripp]

Hi @james-traxler_1560 ,

Thanks for testing that

It will be released in v6.0.16, which is planned to ship on Friday June 10th.

Cheers,
Alana

[james.traxler_1560]

Thank you @atripp !