1. Home
  2. Categories
  3. Support
  4. Docker impossible to push

Welcome to the Inedo Forums! Check out the Forums Guide for help getting started.

If you are experiencing any issues with the forum software, please visit the Contact Form on our website and let us know!

Docker impossible to push

  • I

    Hi,

    We have upgrade our proget from 5.2.32 to 5.3.12 and after 5.3.13.

    Since this upgrade, we cannot push docker images anymore.

    We obtain an error 500.

    A 500 error occurred in registry: Move failed. blobs/sha256/2622b3cbec4c3f908fde9a413e48eca0145887f5c0719a4384d2e862978270b0 already exists.

    URL: http://****.ch/v2/registry/library/autoguaranteefilegenerator/blobs/uploads/0bbc8f8c-5ee6-40d7-a626-52cb7589e2a1?digest=sha256%3A2622b3cbec4c3f908fde9a413e48eca0145887f5c0719a4384d2e862978270b0
    Referrer: (not set)
    User: (unknown)
    User Agent: docker/19.03.12 go/go1.13.10 git-commit/48a66213fe kernel/4.15.0-118-generic os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.12 (linux))
    Stack trace:

    This file doesn't exist on file system or database.

    What is happening ?

    0
  • rhessinger
    inedo-engineer

    Hi @informatique_1703,

    Is this the error from ProGet or from the Docker client. Do you see any other errors in the ProGet diagnostics center when you try to push an image?

    Thanks,
    Rich

    0 I 1 Reply
  • I

    Hi @rhessinger

    This error comes from ProGet.

    The ProGet diagnostics center don't show another error

    0
  • rhessinger
    inedo-engineer

    Hi @informatique_1703,

    Does this happen for all images or just the one image? Do you see any errors in the console from docker?

    Thanks,
    Rich

    0 I 1 Reply
  • I

    Hi @rhessinger ,

    For all images :(

    No more errors in console...

    0
  • atripp
    inedo-engineer

    Hello,

    We aren't getting any other reports of this problem, and it's very unusual to get a 500 error that isn't logged. So it's really hard to guess what the problem could be.

    Can you enable the "Feed Error Logging" to increase the logging? https://docs.inedo.com/docs/proget/installation/diagnostic-center

    Alternatively, you set-up a brand new ProGet instance, then create a Docker feed, and try pushing it?

    Thanks.
    Alana

    0 I 1 Reply
  • I

    Hi @atripp ,

    After change logs level, there are not more information :(

    Logged: 10/12/2020 15:48:33
    Level: Debug
    Category: Feed
    Message: A 500 error occurred in registry: Move failed. blobs/sha256/2622b3cbec4c3f908fde9a413e48eca0145887f5c0719a4384d2e862978270b0 already exists.
    Details: URL: http://***.ch/v2/registry/library/autoguaranteefilegenerator/blobs/uploads/26220be6-794f-4d18-b41a-7dd1c185d63f?digest=sha256%3A2622b3cbec4c3f908fde9a413e48eca0145887f5c0719a4384d2e862978270b0
    Referrer: (not set)
    User: (unknown)
    User Agent: docker/19.03.12 go/go1.13.10 git-commit/48a66213fe kernel/4.15.0-118-generic os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.12 (linux))
    Stack trace:

    If we create a new docker feed it works ...
    It is possible de transfert all packages to the second feed ?

    Thanks

    0
  • rhessinger
    inedo-engineer

    Hi @informatique_1703,

    Currently, the only way to transfer all docker images automatically between feeds is by using feed replication, which requires an enterprise license.

    One manual option is to promote the image from your old feed to your new feed. You would need to do that on each image manually though.

    Is your ProGet instacnce installed on Windows or are you using a Docker image based version of ProGet?

    Thanks,
    Rich

    0 I 1 Reply
  • I

    Hi @rhessinger,

    We using the Docker Version.

    0
  • rhessinger
    inedo-engineer

    Hi @informatique_1703,

    Would you be able to switch to the progetmono image and see if you still have this issue? I would like to rule out .NET Core as a cause.

    Thanks,
    Rich

    0 I 1 Reply
  • I

    Hi @rhessinger,

    Same error with progetmono and no more information :(

    0
  • rhessinger
    inedo-engineer

    Hi @informatique_1703,

    Are you using any docker connectors on this feed? If so, have you cleared the cache on these recently? Also, if you create a new feed, then push the image, does that error happen?

    Thanks,
    Rich

    0 I 1 Reply
  • I

    Hi @rhessinger,

    We have try to clear the cache of the docker connector but have always the error.

    If we create a new feed it works...

    0
  • rhessinger
    inedo-engineer

    Hi @informatique_1703,

    I think I see the issue here. There was a bug, PG-1832, in the clear docker connector cache that was patched in ProGet 5.3.13. The issue that was happening is that the clear cache would fail when a blob was shared between a chached image and a local image. The side affect is that it would delete the blob from the storage location, but it fail to remove the reference from the database. This causes our API to think the blob exists but then fails to find it on the disk.

    This is why a new feed will work because you are pushing the new layer up that doesn't already exist in the feed. If you copy the blob named 2622b3cbec4c3f908fde9a413e48eca0145887f5c0719a4384d2e862978270b0 from the new feed storage location to the old feed storage location, then most likely the push will work. You can find the storage location by checking the Storage.PackagesRootPath, Storage.DockerRepositoryLibrary, and Storage.DockerBlobStorageLibrary in Administration -> Advanced Settings. You may also need to check in the Manage Feed settings for a registry level storage or if your registry is using Shared Storage.

    Thanks,
    Rich

    0 I 1 Reply
  • I

    Hi @rhessinger,

    Same error in 5.3.13 and cache cleared...

    0
  • rhessinger
    inedo-engineer

    Hi @informatique_1703,

    I'm sorry for the confusion. If the clear cache was ran in 5.3.12 or lower, that may have removed the shared blob from your storage location, but left the link in the database. Upgrading to 5.3.13 removes the possibility of deleting a blob that is still in use from your storage location. Can you verify if 2622b3cbec4c3f908fde9a413e48eca0145887f5c0719a4384d2e862978270b0 exists in your storage location? You can also see by going to an image that has that blob layer and looking at the 'Layers` tab. If it has a red exclimation mark next to the layer, then it was most likely removed from the storage location.

    Thanks,
    Rich

    0 I 1 Reply
  • I

    Hi @rhessinger,

    This issue is already present in 5.2.32 ?

    This blob is not present on the storage location and in the database...

    0
  • rhessinger
    inedo-engineer

    Hi @informatique_1703,

    Thanks for clarifying that for me. The issue did exist in 5.2.32, but the scenario to recreate it was different. Now that I understand that the blob is not in the database (I apologize for missing that before), I don't think the clear cache issue caused this. Just to verify, when you searched for the blob digest in the database, did you limit it only to that FeedId? Or did you just search for any blobs with that digest? Could you send me the SQL query you used? Also, can you check to see if this feed is configured to use shared storage? You can find that by navigating to the Manage Feed page and selecting the Storage & Retention tab, then click the Configure link to the right of Blob Storage.

    Thanks,
    Rich

    0
  • A

    I'm also getting this same error was on 5.3.12 when it started updated to 5.3.13 both mono and dotnet are not working with the Error

     Logged: 10/17/2020 13:12:50
 Level: Warning
 Category: Feed
 Message: A 500 error occurred in lcc-registry: Move failed. blobs/sha256/96a20c710d154a49eff959c842039d3969db46ef3904cfdf9a558f17ebcf2bee already exists.
 Details: URL: http://[URL]/v2/[Registry]/library/[Image]/blobs/uploads/6469c2a2-b41c-459e-8fb5-0d63d2f63c4a?digest=sha256:96a20c710d154a49eff959c842039d3969db46ef3904cfdf9a558f17ebcf2bee
Referrer: (not set)
User: Admin
User Agent: docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/3.10.0-957.el7.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.6 \(linux\))
Stack trace:

    Please see error messages above
    also checked the directory and the quoted files doesn't exisit

    0
  • rhessinger
    inedo-engineer

    Hi @aneequaye_1868,

    Would you be able to run an ls -l to check your folder permissions? I have seen some recent issues that linux reports back some weird errors when a folder doesn't have the proper read/write/delete permissions.

    Thanks,
    Rich

    0
  • rhessinger
    inedo-engineer

    Hi @aneequaye_1868 & @informatique_1703,

    I may have found a potential cause for this. Can you verify a feed folder exists in your /proget-packages/? You should see proget-packages/.docker/common and/or /proget-packages/.docker/F<Feed ID> (ex: proget-packages/.docker/F1).

    Thanks,
    Rich

    0 A 1 Reply
  • A

    Hello @rhessinger
    Please did the default path for the packages change for docker images as according to the doc it supposed to reside in /var/proget/packages as such that's where the volume on the docker has been mapped.

    Also the ./.docker/F2 folder exists in the /var/proget/packages/ Folder

    0
  • rhessinger
    inedo-engineer

    Hi @aneequaye_1868,

    I apologize, I was speaking from the mapped folder on the Docker host. Inside the container, it is still /var/proget/packages. Does this still happen on the latest version of ProGet? We did make some changes to how the headers are returned in the latest version, especially for feeds that require authentication.

    Thanks,
    Rich

    0
Log in to reply
 
Inedo Website Home • Support Home • Code of Conduct • Forums Guide • Documentation