Hi @dean-houston ,
Thanks for your response, at the end, we come to the same conclusion that something went wrong during the upgrade and some of the configuration blobs got deleted from the disk and also from the database table. Luckily, we have the backup, but we need to figure out which images are corrupted and need to be recovered.
Gabor
Recently upgraded ProGet free version from 5.xx to latest 24.xx. New docker images can be pushed and pulled without any problem, BUT none of the docker images which were pushed to proget version 5.xx can be pulled via 24.xx anymore.
By examining image manifests and proget server console log, it turned out that although these old images' manifest files contain config entries, the referenced config blobs do not exist in the storage, therefore all image pull fail with: "error pulling image configuration: download failed after attempts=1: unknown blob".
The questions are:
why these config blobs cannot be found in the storage if needed for pull?
how could docker pull work in case of proget 5.xx without those files?
Thx,
By migrating to ProGet to use as the NuGet server company wide, (as more and more developers switched to the new NuGet server) we were getting strange build errors relating to references coming from NuGet packages.
First, we tried to handle the issue by clearing all local NuGet caches, upgrading/downgrading nuget.exe, but later, it turned out that the source of the problem is, that some of the NuGet package folders in local cache do not contain the correct NuGet package, e.g. folder A.B.C/1.2.3 contains the content of X.Y.Z-1.1.0.nupkg?!?
At the end, we wrote a simple test application to download random .nupkg files parallely from ProGet server and verify the retrieved content of each .nupkg files, the result was disastrous: by trying to download 1000 random .nupkg parallely, only 89 packages were correct, the rest, 911 packages contained .nupkg of an other package being requested at the same time!
Obviously, there is a serious concurreny issue inside ProGet server, that by requesting A.B.C.nupkg, then X.Y.Z.nupkg is received.
This issue was first detected in an environment as:
OS = CentOS, ProGet = 5.2.7, Docker = 19.03.0, DB = MSSQL 2016 SP2, NuGet packages = 350 pieces and ~25K versions alltogether
And confirmed in following environment:
OS = CentOS, ProGet = 5.2.14, Docker = 19.03.0, DB = MSSQL 2016 SP2, NuGet packages = 350 pieces and ~25K versions alltogether
OS = Win10, ProGet = 5.2.14, Docker = 19.03.4, DB = MSSQL 2017 Docker, NuGet packages = 1K
OS = Win10, ProGet = 5.2.14 (both self-hosted and IIS hosted), DB = MSSQL 2017 Docker, NuGet packages = 1K