Bug: ProGet caches invalid Docker layer blob, keeps serving it

cyril

An issue I unfortunately have been running into recently; we're using ProGet with connectors to other registries. For an unknown reason (I suspect some network issues upstream), large-ish images (e.g., 400mb) that are pulled by ProGet through its connector get dropped. Likely due to the download just taking longer, therefore more opportunity for the connection to drop. Timeouts and related settings are within reasonable amounts.

In any case, the result is that ProGet shows that image as available and cached, but when pulled by a client, it fails due to size mismatch:

docker pull proget.myorg.com/...
...
e28c9cbe4ea2: Download complete
short read: expected 64432621 bytes but got 38190973: unexpected EOF

The only way I've found to fix this (manually) is to stop clients from requesting the multiple images that might share that layer, then delete the image, then delete the metadata cache, and finally run the Docker/Feed cleanup executions. Feed integrity shows green, and running the cleanup without deletion also had no effect. Having to stop the downstream clients was also a concern: I was otherwise racing before they requested the image again & repopulated the (invalid) cache.

It essentially seems like ProGet is keeping partial/corrupted blobs, and then serving them as though they were correct, with no check and no mechanism to resolve itself. Is there anything I can do here?

(Additional context: running 2026.14 Linux image)

atripp

Hi @cyril ,

It's hard to really say but it could be a client/caching issue.

I can describe ProGet's behavior:

when adding a blob to a feed, ProGet first requests an exclusive write lock on a file, writes the entirety of the stream to that file while hashing it, confirms the hash, then writes a database entry indicating the successful file write and size
when downloading a blob from a connector, a race condition may occur when adding the cached file to the feed; this is ignored, and the remote stream is returned
when performing a Docker feed scan, ProGet will recompute the hash of each blob and report on issues
when performing a reindex (with deletion), ProGet will perform another hash recompute and delete mismatched blobs

That said, aside from OS-level interference -- which happens A LOT due to endpoint monitoring tools -- we're not aware of any conditions in which a blob is truncated or fails to write. I can't imagine how the integrity check would fail to detect a truncated blob.

The next time you see an error like this, I would check the content on ProGet server itself. For example, when you see this:

e28c9cbe4ea2: Download complete
short read: expected 64432621 bytes but got 38190973: unexpected EOF

Track down the e28c9cbe4ea2 blobfile in the package store. It should have a hash that matches the digest.

Separately, you can test ProGet's behavior by going to the file store and manipulating a blob file. For example, rename abcd1234 to zzz_abcd1234 then copy some other blob and name it abcd1234. That will always fail a hashcheck.

Thanks,
Alana

cyril

@atripp thanks, I'll investigate some more in that case. In case it might trigger other thoughts: I forgot to mention we're using cloud storage (S3, on premise so still very high throughput), with common blob storage disabled.