Bug: ProGet caches invalid Docker layer blob, keeps serving it

cyril

An issue I unfortunately have been running into recently; we're using ProGet with connectors to other registries. For an unknown reason (I suspect some network issues upstream), large-ish images (e.g., 400mb) that are pulled by ProGet through its connector get dropped. Likely due to the download just taking longer, therefore more opportunity for the connection to drop. Timeouts and related settings are within reasonable amounts.

In any case, the result is that ProGet shows that image as available and cached, but when pulled by a client, it fails due to size mismatch:

docker pull proget.myorg.com/...
...
e28c9cbe4ea2: Download complete
short read: expected 64432621 bytes but got 38190973: unexpected EOF

The only way I've found to fix this (manually) is to stop clients from requesting the multiple images that might share that layer, then delete the image, then delete the metadata cache, and finally run the Docker/Feed cleanup executions. Feed integrity shows green, and running the cleanup without deletion also had no effect. Having to stop the downstream clients was also a concern: I was otherwise racing before they requested the image again & repopulated the (invalid) cache.

It essentially seems like ProGet is keeping partial/corrupted blobs, and then serving them as though they were correct, with no check and no mechanism to resolve itself. Is there anything I can do here?

(Additional context: running 2026.14 Linux image)

atripp

Hi @cyril ,

It's hard to really say but it could be a client/caching issue.

I can describe ProGet's behavior:

when adding a blob to a feed, ProGet first requests an exclusive write lock on a file, writes the entirety of the stream to that file while hashing it, confirms the hash, then writes a database entry indicating the successful file write and size
when downloading a blob from a connector, a race condition may occur when adding the cached file to the feed; this is ignored, and the remote stream is returned
when performing a Docker feed scan, ProGet will recompute the hash of each blob and report on issues
when performing a reindex (with deletion), ProGet will perform another hash recompute and delete mismatched blobs

That said, aside from OS-level interference -- which happens A LOT due to endpoint monitoring tools -- we're not aware of any conditions in which a blob is truncated or fails to write. I can't imagine how the integrity check would fail to detect a truncated blob.

The next time you see an error like this, I would check the content on ProGet server itself. For example, when you see this:

e28c9cbe4ea2: Download complete
short read: expected 64432621 bytes but got 38190973: unexpected EOF

Track down the e28c9cbe4ea2 blobfile in the package store. It should have a hash that matches the digest.

Separately, you can test ProGet's behavior by going to the file store and manipulating a blob file. For example, rename abcd1234 to zzz_abcd1234 then copy some other blob and name it abcd1234. That will always fail a hashcheck.

Thanks,
Alana

cyril

@atripp thanks, I'll investigate some more in that case. In case it might trigger other thoughts: I forgot to mention we're using cloud storage (S3, on premise so still very high throughput), with common blob storage disabled.

cyril

Hi @atripp,

I've encountered the issue again, and have been able to track down the issue. Seems related to S3.

Pulling the image:

failed to pull and unpack image "<proget>/ghcrio-proxy/fluxcd/image-reflector-controller:v1.2.5":
short read: expected 37867329 bytes but got 32624449: unexpected EOF

ProGet did not log any errors, and then serves the blob happily with a 200;

Request finished HTTP/1.1 GET http://<proget>/v2/ghcrio-proxy/fluxcd/image-reflector-controller/blobs/sha256:266ebec216df42de9ed7c4d07422168f3329eecf923306036386404f10c5a0f8 - 200 32624449 application/vnd.oci.image.layer.v1.tar+gzip 223.6409ms

So ProGet is serving 32624449 bytes, which is what the client rejects.

expected: 37867329 = 7 x 5242880 + 1167169   (7 full parts + final part)
stored:   32624449 = 6 x 5242880 + 1167169   (6 full parts + final part)
missing:   5242880 = exactly one full 5 MiB part

It seems ProGet failed to store one part of a multipart upload to S3?

In case it can help speed things up, the usual AI assistants came up with these findings:

S3WriteStream.cs in inedox-aws (master, last changed in 97a95a44).

Parts upload in the background as a task chain built in CheckAndBeginBackgroundUpload (L192-L195). Each part is attached with ContinueWith(_ => ...), which runs whether or not the previous task faulted, and the previous task is never checked. If UploadPartAsync for part N throws (for example after the SDK's retries run out), the exception is lost. Part N+1 still uploads, and uploadTask then reflects only the last link in the chain.

CompleteUploadAsync awaits that task (L93-L94), uploads the final part and calls CompleteMultipartUploadAsync with a part list that lacks part N (L115-L140). S3 accepts non-contiguous part numbers, so the upload succeeds with an object exactly one part short and no error is raised. ProGet then stores and serves the blob under its digest.

Write(byte[], int, int) and WriteAsync(byte[], int, int, CancellationToken) ignore offset and always read from index 0 (L41, L46). Any caller that writes from a non-zero offset stores the wrong bytes at the correct length.

Edit: confirmed this is the (missing part) issue: I pulled from my ProGet instance and directly from the upstream, and compared the blobs. The ProGet blob is missing "part 2" of the blob exactly.