After upgrading ProGet, docker images cannot be pulled because missing configuration blob

ghollosy_9163

Recently upgraded ProGet free version from 5.xx to latest 24.xx. New docker images can be pushed and pulled without any problem, BUT none of the docker images which were pushed to proget version 5.xx can be pulled via 24.xx anymore.
By examining image manifests and proget server console log, it turned out that although these old images' manifest files contain config entries, the referenced config blobs do not exist in the storage, therefore all image pull fail with: "error pulling image configuration: download failed after attempts=1: unknown blob".

The questions are:
why these config blobs cannot be found in the storage if needed for pull?
how could docker pull work in case of proget 5.xx without those files?

Thx,

dean-houston

Hi @ghollosy_9163,

I'm afraid this will not be an easy problem to troubleshoot; there were some major changes in ProGet 5.3 with regards to Docker feeds, so it's possible that there was a data migration problem during the upgrade process.

Ultimately it looks like your container configuration files aren't there. I don't know why, but I'll try to point you to where/how they are stored.

Docker blobs are stored on disk, and you'll see files named by their hash. Based your screenshot, it'd be a file like bf3c18259..... These files are also indexed in the database, in the DockerBlobs table.

If the files aren't anywhere on disk, then check backups. It's possible they got accidently deleted, and you can just add them back.

If the files are missing on disk, then they are probably gone for ever. If they are not in the database, then perhaps something happened during the migration? I really don't know. It's possible to insert them in the database, but we don't have an automatic way to do this - basically you'd need to write a script.

-- Dean

ghollosy_9163

Hi @dean-houston ,

Thanks for your response, at the end, we come to the same conclusion that something went wrong during the upgrade and some of the configuration blobs got deleted from the disk and also from the database table. Luckily, we have the backup, but we need to figure out which images are corrupted and need to be recovered.

Gabor