Bug: duplicate docker manifests on connected feed when upstream tag updated

[mayorovp_3701]

We use Proget to mirror docker hub for faster builds, but sometimes pull from project just fail due to internal server error.

Our configuration contains the feed, and connector for that feed to docker hub. The feed somehow starts to contain two copies of the same manifest with identical digests, and every attempt to pull this image fail due to "two manifests with same digets" error on proget side.

Also our configuration contains gitlab runner that can run two jobs in parallel, and CI config that contains two jobs with same image (docker:cli). And that image is failed to pull each time new version pushed on docker hub.

I was unable to reproduce this failed state (probably wrong timings), but possible steps are like this:

1. configure feed and connector,
2. pull some image with floating tag from feed,
3. update image at upstream (or wait for image to be updated by maintainers),
4. pull that image again by two clients at the same time

Expected result:

New image cached at proget, and pulled by clients

Actual result:

New image cached at proget twice, but cannot be pulled by clients.

[atripp]

Hi @mayorovp_3701

This one is tricky and I'm afraid I cannot reproduce this error either.... but I think I can see how it's possible.

First off, it's definitely possible for two processes to attempt to add the same manifest to the feed at the same time, and I might expect that during a parallel run like you describe.

However, there is data integrity constraint (trigger) to design this from happening. This trigger should yield a Image_Digest must be unique across the containing feed error.

But, looking at the trigger code for PostgreSQL, it looks like we aren't using an advisory lock to completely prevent a race condition.

Can you confirm that you're using PostgreSQL?

Thanks,
Alana

[mayorovp_3701]

This post is deleted!

[mayorovp_3701]

@atripp Yes, we use inner PostgreSQL.

[atripp]

Hi @mayorovp_3701 ,

Thanks for confirming; we can will try to get this fixed in the next maintenance release via PG-3139 -- the underlying issue is most likely a race condition on that trigger I mentioned, so we're going to fix it by adding an advisory lock.

It will not remove the duplicate content, but in theory deleting the image will.

Thanks,
Alana