Docker impossible to push

informatique_1703

For all images :(

No more errors in console...

atripp

Hello,

We aren't getting any other reports of this problem, and it's very unusual to get a 500 error that isn't logged. So it's really hard to guess what the problem could be.

Can you enable the "Feed Error Logging" to increase the logging? https://docs.inedo.com/docs/proget/installation/diagnostic-center

Alternatively, you set-up a brand new ProGet instance, then create a Docker feed, and try pushing it?

Thanks.
Alana

informatique_1703

Hi @atripp ,

After change logs level, there are not more information :(

Logged: 10/12/2020 15:48:33
Level: Debug
Category: Feed
Message: A 500 error occurred in registry: Move failed. blobs/sha256/2622b3cbec4c3f908fde9a413e48eca0145887f5c0719a4384d2e862978270b0 already exists.
Details: URL: http://***.ch/v2/registry/library/autoguaranteefilegenerator/blobs/uploads/26220be6-794f-4d18-b41a-7dd1c185d63f?digest=sha256%3A2622b3cbec4c3f908fde9a413e48eca0145887f5c0719a4384d2e862978270b0
Referrer: (not set)
User: (unknown)
User Agent: docker/19.03.12 go/go1.13.10 git-commit/48a66213fe kernel/4.15.0-118-generic os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.12 (linux))
Stack trace:

If we create a new docker feed it works ...
It is possible de transfert all packages to the second feed ?

Thanks

rhessinger

Hi @informatique_1703,

Currently, the only way to transfer all docker images automatically between feeds is by using feed replication, which requires an enterprise license.

One manual option is to promote the image from your old feed to your new feed. You would need to do that on each image manually though.

Is your ProGet instacnce installed on Windows or are you using a Docker image based version of ProGet?

Thanks,
Rich

informatique_1703

Hi @rhessinger,

We using the Docker Version.

rhessinger

Hi @informatique_1703,

Would you be able to switch to the progetmono image and see if you still have this issue? I would like to rule out .NET Core as a cause.

Thanks,
Rich

informatique_1703

Hi @rhessinger,

Same error with progetmono and no more information :(

rhessinger

Hi @informatique_1703,

Are you using any docker connectors on this feed? If so, have you cleared the cache on these recently? Also, if you create a new feed, then push the image, does that error happen?

Thanks,
Rich

informatique_1703

Hi @rhessinger,

We have try to clear the cache of the docker connector but have always the error.

If we create a new feed it works...

rhessinger

Hi @informatique_1703,

I think I see the issue here. There was a bug, PG-1832, in the clear docker connector cache that was patched in ProGet 5.3.13. The issue that was happening is that the clear cache would fail when a blob was shared between a chached image and a local image. The side affect is that it would delete the blob from the storage location, but it fail to remove the reference from the database. This causes our API to think the blob exists but then fails to find it on the disk.

This is why a new feed will work because you are pushing the new layer up that doesn't already exist in the feed. If you copy the blob named 2622b3cbec4c3f908fde9a413e48eca0145887f5c0719a4384d2e862978270b0 from the new feed storage location to the old feed storage location, then most likely the push will work. You can find the storage location by checking the Storage.PackagesRootPath, Storage.DockerRepositoryLibrary, and Storage.DockerBlobStorageLibrary in Administration -> Advanced Settings. You may also need to check in the Manage Feed settings for a registry level storage or if your registry is using Shared Storage.

Thanks,
Rich

informatique_1703

Hi @rhessinger,

Same error in 5.3.13 and cache cleared...

rhessinger

Hi @informatique_1703,

I'm sorry for the confusion. If the clear cache was ran in 5.3.12 or lower, that may have removed the shared blob from your storage location, but left the link in the database. Upgrading to 5.3.13 removes the possibility of deleting a blob that is still in use from your storage location. Can you verify if 2622b3cbec4c3f908fde9a413e48eca0145887f5c0719a4384d2e862978270b0 exists in your storage location? You can also see by going to an image that has that blob layer and looking at the 'Layers` tab. If it has a red exclimation mark next to the layer, then it was most likely removed from the storage location.

Thanks,
Rich

informatique_1703

Hi @rhessinger,

This issue is already present in 5.2.32 ?

This blob is not present on the storage location and in the database...

rhessinger

Hi @informatique_1703,

Thanks for clarifying that for me. The issue did exist in 5.2.32, but the scenario to recreate it was different. Now that I understand that the blob is not in the database (I apologize for missing that before), I don't think the clear cache issue caused this. Just to verify, when you searched for the blob digest in the database, did you limit it only to that FeedId? Or did you just search for any blobs with that digest? Could you send me the SQL query you used? Also, can you check to see if this feed is configured to use shared storage? You can find that by navigating to the Manage Feed page and selecting the Storage & Retention tab, then click the Configure link to the right of Blob Storage.

Thanks,
Rich

aneequaye_1868

I'm also getting this same error was on 5.3.12 when it started updated to 5.3.13 both mono and dotnet are not working with the Error

 Logged: 10/17/2020 13:12:50
 Level: Warning
 Category: Feed
 Message: A 500 error occurred in lcc-registry: Move failed. blobs/sha256/96a20c710d154a49eff959c842039d3969db46ef3904cfdf9a558f17ebcf2bee already exists.
 Details: URL: http://[URL]/v2/[Registry]/library/[Image]/blobs/uploads/6469c2a2-b41c-459e-8fb5-0d63d2f63c4a?digest=sha256:96a20c710d154a49eff959c842039d3969db46ef3904cfdf9a558f17ebcf2bee
Referrer: (not set)
User: Admin
User Agent: docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/3.10.0-957.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.6 \(linux\))
Stack trace:

Please see error messages above
also checked the directory and the quoted files doesn't exisit

rhessinger

Hi @aneequaye_1868,

Would you be able to run an ls -l to check your folder permissions? I have seen some recent issues that linux reports back some weird errors when a folder doesn't have the proper read/write/delete permissions.

Thanks,
Rich

rhessinger

Hi @aneequaye_1868 & @informatique_1703,

I may have found a potential cause for this. Can you verify a feed folder exists in your /proget-packages/? You should see proget-packages/.docker/common and/or /proget-packages/.docker/F<Feed ID> (ex: proget-packages/.docker/F1).

Thanks,
Rich

aneequaye_1868

Hello @rhessinger
Please did the default path for the packages change for docker images as according to the doc it supposed to reside in /var/proget/packages as such that's where the volume on the docker has been mapped.

Also the ./.docker/F2 folder exists in the /var/proget/packages/ Folder

rhessinger

Hi @aneequaye_1868,

I apologize, I was speaking from the mapped folder on the Docker host. Inside the container, it is still /var/proget/packages. Does this still happen on the latest version of ProGet? We did make some changes to how the headers are returned in the latest version, especially for feeds that require authentication.

Thanks,
Rich