Inedo Community Forums Forums
    • Recent
    • Tags
    • Popular
    • Login
    1. Home
    2. cyril

    Welcome to the Inedo Forums! Check out the Forums Guide for help getting started.

    If you are experiencing any issues with the forum software, please visit the Contact Form on our website and let us know!

    C Offline
    • Profile
    • Following 0
    • Followers 0
    • Topics 5
    • Posts 14
    • Groups 0

    cyril

    @cyril

    1
    Profile views
    14
    Posts
    0
    Followers
    0
    Following
    Joined
    Last Online

    cyril Unfollow Follow

    Latest posts made by cyril

    • RE: Bug: ProGet caches invalid Docker layer blob, keeps serving it

      @atripp small follow-up, I asked our platform team for some help investigating on their end, they observed the issue from the S3 side:

      Between 13:40:50 and 13:41:07 UTC, 16 multipart UploadPart PUTs from ProGet ([redacted]) to [redacted]/ghcr-proxy/blobs/... returned HTTP 400.

      The 400s came from our ingress-nginx, not [S3 solution]. [S3 solution] has no record of them. The requests never reached an upstream, and the bodies were shorter than a full ~5.25 MB part, for example 0.45, 2.0 and 3.2 MB. That is what nginx usually logs when the client closes the connection partway through sending the body.

      Four uploads were hit at about the same moment, and most failed parts succeeded on retry.
      One didn’t: sha256:266ebec216df42de9ed7c4d07422168f3329eecf923306036386404f10c5a0f8. Part 2 failed three times with no success after.

      ProGet still sent CompleteMultipartUpload at 13:40:53 and [S3 solution] returned 200. [S3 solution] accepts a completion that omits a part. I expect that object is missing part 2, which would explain the truncated blob.

      posted in Support
      C
      cyril
    • RE: Bug: ProGet caches invalid Docker layer blob, keeps serving it

      Hi @atripp,

      I've encountered the issue again, and have been able to track down the issue. Seems related to S3.

      Pulling the image:

      failed to pull and unpack image "<proget>/ghcrio-proxy/fluxcd/image-reflector-controller:v1.2.5":
      short read: expected 37867329 bytes but got 32624449: unexpected EOF
      

      ProGet did not log any errors, and then serves the blob happily with a 200;

      Request finished HTTP/1.1 GET http://<proget>/v2/ghcrio-proxy/fluxcd/image-reflector-controller/blobs/sha256:266ebec216df42de9ed7c4d07422168f3329eecf923306036386404f10c5a0f8 - 200 32624449 application/vnd.oci.image.layer.v1.tar+gzip 223.6409ms
      

      So ProGet is serving 32624449 bytes, which is what the client rejects.

      expected: 37867329 = 7 x 5242880 + 1167169   (7 full parts + final part)
      stored:   32624449 = 6 x 5242880 + 1167169   (6 full parts + final part)
      missing:   5242880 = exactly one full 5 MiB part
      

      It seems ProGet failed to store one part of a multipart upload to S3?


      In case it can help speed things up, the usual AI assistants came up with these findings:

      S3WriteStream.cs in inedox-aws (master, last changed in 97a95a44).

      Parts upload in the background as a task chain built in CheckAndBeginBackgroundUpload (L192-L195). Each part is attached with ContinueWith(_ => ...), which runs whether or not the previous task faulted, and the previous task is never checked. If UploadPartAsync for part N throws (for example after the SDK's retries run out), the exception is lost. Part N+1 still uploads, and uploadTask then reflects only the last link in the chain.

      CompleteUploadAsync awaits that task (L93-L94), uploads the final part and calls CompleteMultipartUploadAsync with a part list that lacks part N (L115-L140). S3 accepts non-contiguous part numbers, so the upload succeeds with an object exactly one part short and no error is raised. ProGet then stores and serves the blob under its digest.


      Write(byte[], int, int) and WriteAsync(byte[], int, int, CancellationToken) ignore offset and always read from index 0 (L41, L46). Any caller that writes from a non-zero offset stores the wrong bytes at the correct length.


      Edit: confirmed this is the (missing part) issue: I pulled from my ProGet instance and directly from the upstream, and compared the blobs. The ProGet blob is missing "part 2" of the blob exactly.

      posted in Support
      C
      cyril
    • RE: Bug: ProGet caches invalid Docker layer blob, keeps serving it

      @atripp thanks, I'll investigate some more in that case. In case it might trigger other thoughts: I forgot to mention we're using cloud storage (S3, on premise so still very high throughput), with common blob storage disabled.

      posted in Support
      C
      cyril
    • Bug: ProGet caches invalid Docker layer blob, keeps serving it

      An issue I unfortunately have been running into recently; we're using ProGet with connectors to other registries. For an unknown reason (I suspect some network issues upstream), large-ish images (e.g., 400mb) that are pulled by ProGet through its connector get dropped. Likely due to the download just taking longer, therefore more opportunity for the connection to drop. Timeouts and related settings are within reasonable amounts.

      In any case, the result is that ProGet shows that image as available and cached, but when pulled by a client, it fails due to size mismatch:

      docker pull proget.myorg.com/...
      ...
      e28c9cbe4ea2: Download complete
      short read: expected 64432621 bytes but got 38190973: unexpected EOF
      

      The only way I've found to fix this (manually) is to stop clients from requesting the multiple images that might share that layer, then delete the image, then delete the metadata cache, and finally run the Docker/Feed cleanup executions. Feed integrity shows green, and running the cleanup without deletion also had no effect. Having to stop the downstream clients was also a concern: I was otherwise racing before they requested the image again & repopulated the (invalid) cache.

      It essentially seems like ProGet is keeping partial/corrupted blobs, and then serving them as though they were correct, with no check and no mechanism to resolve itself. Is there anything I can do here?

      (Additional context: running 2026.14 Linux image)

      posted in Support
      C
      cyril
    • RE: Support for setting Debian feed distribution/components via pgutil / API

      @dean-houston any chance I could suggest the same be done/added for other feed types & connectors?

      Options that show up under "Advanced" for connector configuration (e.g., Python feed's "simple" toggle, or library toggle for Docker), and e.g., the per-feed Settings panes (e.g., Docker Settings, NuGet settings). All our use cases have involved tweaking one of these settings, so it's a bit unfortunate to have gotten so close to the finish line in automation!

      posted in Support
      C
      cyril
    • RE: Bug: Configuring ProGet via the API/pgutil, fails on (certain) type conversions

      Here too, thanks for the quick fix :-)

      posted in Support
      C
      cyril
    • RE: Support for setting Debian feed distribution/components via pgutil / API

      Thanks for the quick fix!

      posted in Support
      C
      cyril
    • Bug: Configuring ProGet via the API/pgutil, fails on (certain) type conversions

      Issue is quite straightforward:

      root@proget:/usr/local/proget# pgutil settings set --name=Web.ConcurrentRequestLimit --value=200
      Server responded with InternalServerError (500): Object of type 'System.String' cannot be converted to type 'System.Nullable`1[System.Int32]'.
      

      I'm able to set the same setting via the page in Administration > HTTP/S & Certificate Settings, however I do see that it's (seemingly) persisted alongside the rest of the configuration rows. I'm guessing there's some predefined list that maps types, that this value doesn't show up in?

      Assuming making all values available directly isn't supported, maybe adding a --type or --int flag to pgutil would do the trick (and --bool, --string respectively)? This would be consistent with other tools I've used.

      posted in Support
      C
      cyril
    • RE: Caching ProGet feeds in S3 and/or auto-creating local repository

      @stevedennis thanks, that did the trick! I

      posted in Support
      C
      cyril
    • RE: Caching ProGet feeds in S3 and/or auto-creating local repository

      @stevedennis I didn't explicitly enable it, but it does seem to be enabled on those feeds indeed.. I did indeed use the API / pgutil.

      I get the following when trying to click 'change'.
      Screenshot 2026-10-23 at 16.00.23.png

      Only seems to affect Docker feeds; I've got other types of feeds (e.g., Helm) that I also configured; those don't show the same error when opening the GUI to edit.

      posted in Support
      C
      cyril