Hi @atripp,
I've encountered the issue again, and have been able to track down the issue. Seems related to S3.
Pulling the image:
failed to pull and unpack image "<proget>/ghcrio-proxy/fluxcd/image-reflector-controller:v1.2.5":
short read: expected 37867329 bytes but got 32624449: unexpected EOF
ProGet did not log any errors, and then serves the blob happily with a 200;
Request finished HTTP/1.1 GET http://<proget>/v2/ghcrio-proxy/fluxcd/image-reflector-controller/blobs/sha256:266ebec216df42de9ed7c4d07422168f3329eecf923306036386404f10c5a0f8 - 200 32624449 application/vnd.oci.image.layer.v1.tar+gzip 223.6409ms
So ProGet is serving 32624449 bytes, which is what the client rejects.
expected: 37867329 = 7 x 5242880 + 1167169 (7 full parts + final part)
stored: 32624449 = 6 x 5242880 + 1167169 (6 full parts + final part)
missing: 5242880 = exactly one full 5 MiB part
It seems ProGet failed to store one part of a multipart upload to S3?
In case it can help speed things up, the usual AI assistants came up with these findings:
S3WriteStream.cs in inedox-aws (master, last changed in 97a95a44).
Parts upload in the background as a task chain built in CheckAndBeginBackgroundUpload (L192-L195). Each part is attached with ContinueWith(_ => ...), which runs whether or not the previous task faulted, and the previous task is never checked. If UploadPartAsync for part N throws (for example after the SDK's retries run out), the exception is lost. Part N+1 still uploads, and uploadTask then reflects only the last link in the chain.
CompleteUploadAsync awaits that task (L93-L94), uploads the final part and calls CompleteMultipartUploadAsync with a part list that lacks part N (L115-L140). S3 accepts non-contiguous part numbers, so the upload succeeds with an object exactly one part short and no error is raised. ProGet then stores and serves the blob under its digest.
Write(byte[], int, int) and WriteAsync(byte[], int, int, CancellationToken) ignore offset and always read from index 0 (L41, L46). Any caller that writes from a non-zero offset stores the wrong bytes at the correct length.
Edit: confirmed this is the (missing part) issue: I pulled from my ProGet instance and directly from the upstream, and compared the blobs. The ProGet blob is missing "part 2" of the blob exactly.