Inedo Community Forums Forums
    • Recent
    • Tags
    • Popular
    • Login
    1. Home
    2. cyril
    3. Posts

    Welcome to the Inedo Forums! Check out the Forums Guide for help getting started.

    If you are experiencing any issues with the forum software, please visit the Contact Form on our website and let us know!

    C Offline
    • Profile
    • Following 0
    • Followers 0
    • Topics 5
    • Posts 14
    • Groups 0

    Posts

    Recent
    • RE: Bug: ProGet caches invalid Docker layer blob, keeps serving it

      @atripp small follow-up, I asked our platform team for some help investigating on their end, they observed the issue from the S3 side:

      Between 13:40:50 and 13:41:07 UTC, 16 multipart UploadPart PUTs from ProGet ([redacted]) to [redacted]/ghcr-proxy/blobs/... returned HTTP 400.

      The 400s came from our ingress-nginx, not [S3 solution]. [S3 solution] has no record of them. The requests never reached an upstream, and the bodies were shorter than a full ~5.25 MB part, for example 0.45, 2.0 and 3.2 MB. That is what nginx usually logs when the client closes the connection partway through sending the body.

      Four uploads were hit at about the same moment, and most failed parts succeeded on retry.
      One didn’t: sha256:266ebec216df42de9ed7c4d07422168f3329eecf923306036386404f10c5a0f8. Part 2 failed three times with no success after.

      ProGet still sent CompleteMultipartUpload at 13:40:53 and [S3 solution] returned 200. [S3 solution] accepts a completion that omits a part. I expect that object is missing part 2, which would explain the truncated blob.

      posted in Support
      C
      cyril
    • RE: Bug: ProGet caches invalid Docker layer blob, keeps serving it

      Hi @atripp,

      I've encountered the issue again, and have been able to track down the issue. Seems related to S3.

      Pulling the image:

      failed to pull and unpack image "<proget>/ghcrio-proxy/fluxcd/image-reflector-controller:v1.2.5":
      short read: expected 37867329 bytes but got 32624449: unexpected EOF
      

      ProGet did not log any errors, and then serves the blob happily with a 200;

      Request finished HTTP/1.1 GET http://<proget>/v2/ghcrio-proxy/fluxcd/image-reflector-controller/blobs/sha256:266ebec216df42de9ed7c4d07422168f3329eecf923306036386404f10c5a0f8 - 200 32624449 application/vnd.oci.image.layer.v1.tar+gzip 223.6409ms
      

      So ProGet is serving 32624449 bytes, which is what the client rejects.

      expected: 37867329 = 7 x 5242880 + 1167169   (7 full parts + final part)
      stored:   32624449 = 6 x 5242880 + 1167169   (6 full parts + final part)
      missing:   5242880 = exactly one full 5 MiB part
      

      It seems ProGet failed to store one part of a multipart upload to S3?


      In case it can help speed things up, the usual AI assistants came up with these findings:

      S3WriteStream.cs in inedox-aws (master, last changed in 97a95a44).

      Parts upload in the background as a task chain built in CheckAndBeginBackgroundUpload (L192-L195). Each part is attached with ContinueWith(_ => ...), which runs whether or not the previous task faulted, and the previous task is never checked. If UploadPartAsync for part N throws (for example after the SDK's retries run out), the exception is lost. Part N+1 still uploads, and uploadTask then reflects only the last link in the chain.

      CompleteUploadAsync awaits that task (L93-L94), uploads the final part and calls CompleteMultipartUploadAsync with a part list that lacks part N (L115-L140). S3 accepts non-contiguous part numbers, so the upload succeeds with an object exactly one part short and no error is raised. ProGet then stores and serves the blob under its digest.


      Write(byte[], int, int) and WriteAsync(byte[], int, int, CancellationToken) ignore offset and always read from index 0 (L41, L46). Any caller that writes from a non-zero offset stores the wrong bytes at the correct length.


      Edit: confirmed this is the (missing part) issue: I pulled from my ProGet instance and directly from the upstream, and compared the blobs. The ProGet blob is missing "part 2" of the blob exactly.

      posted in Support
      C
      cyril
    • RE: Bug: ProGet caches invalid Docker layer blob, keeps serving it

      @atripp thanks, I'll investigate some more in that case. In case it might trigger other thoughts: I forgot to mention we're using cloud storage (S3, on premise so still very high throughput), with common blob storage disabled.

      posted in Support
      C
      cyril
    • Bug: ProGet caches invalid Docker layer blob, keeps serving it

      An issue I unfortunately have been running into recently; we're using ProGet with connectors to other registries. For an unknown reason (I suspect some network issues upstream), large-ish images (e.g., 400mb) that are pulled by ProGet through its connector get dropped. Likely due to the download just taking longer, therefore more opportunity for the connection to drop. Timeouts and related settings are within reasonable amounts.

      In any case, the result is that ProGet shows that image as available and cached, but when pulled by a client, it fails due to size mismatch:

      docker pull proget.myorg.com/...
      ...
      e28c9cbe4ea2: Download complete
      short read: expected 64432621 bytes but got 38190973: unexpected EOF
      

      The only way I've found to fix this (manually) is to stop clients from requesting the multiple images that might share that layer, then delete the image, then delete the metadata cache, and finally run the Docker/Feed cleanup executions. Feed integrity shows green, and running the cleanup without deletion also had no effect. Having to stop the downstream clients was also a concern: I was otherwise racing before they requested the image again & repopulated the (invalid) cache.

      It essentially seems like ProGet is keeping partial/corrupted blobs, and then serving them as though they were correct, with no check and no mechanism to resolve itself. Is there anything I can do here?

      (Additional context: running 2026.14 Linux image)

      posted in Support
      C
      cyril
    • RE: Support for setting Debian feed distribution/components via pgutil / API

      @dean-houston any chance I could suggest the same be done/added for other feed types & connectors?

      Options that show up under "Advanced" for connector configuration (e.g., Python feed's "simple" toggle, or library toggle for Docker), and e.g., the per-feed Settings panes (e.g., Docker Settings, NuGet settings). All our use cases have involved tweaking one of these settings, so it's a bit unfortunate to have gotten so close to the finish line in automation!

      posted in Support
      C
      cyril
    • RE: Bug: Configuring ProGet via the API/pgutil, fails on (certain) type conversions

      Here too, thanks for the quick fix :-)

      posted in Support
      C
      cyril
    • RE: Support for setting Debian feed distribution/components via pgutil / API

      Thanks for the quick fix!

      posted in Support
      C
      cyril
    • Bug: Configuring ProGet via the API/pgutil, fails on (certain) type conversions

      Issue is quite straightforward:

      root@proget:/usr/local/proget# pgutil settings set --name=Web.ConcurrentRequestLimit --value=200
      Server responded with InternalServerError (500): Object of type 'System.String' cannot be converted to type 'System.Nullable`1[System.Int32]'.
      

      I'm able to set the same setting via the page in Administration > HTTP/S & Certificate Settings, however I do see that it's (seemingly) persisted alongside the rest of the configuration rows. I'm guessing there's some predefined list that maps types, that this value doesn't show up in?

      Assuming making all values available directly isn't supported, maybe adding a --type or --int flag to pgutil would do the trick (and --bool, --string respectively)? This would be consistent with other tools I've used.

      posted in Support
      C
      cyril
    • RE: Caching ProGet feeds in S3 and/or auto-creating local repository

      @stevedennis thanks, that did the trick! I

      posted in Support
      C
      cyril
    • RE: Caching ProGet feeds in S3 and/or auto-creating local repository

      @stevedennis I didn't explicitly enable it, but it does seem to be enabled on those feeds indeed.. I did indeed use the API / pgutil.

      I get the following when trying to click 'change'.
      Screenshot 2026-10-23 at 16.00.23.png

      Only seems to affect Docker feeds; I've got other types of feeds (e.g., Helm) that I also configured; those don't show the same error when opening the GUI to edit.

      posted in Support
      C
      cyril
    • RE: Caching ProGet feeds in S3 and/or auto-creating local repository

      @stevedennis thanks for the quick reply.

      Something seems to going wrong then... For example, I've got a feed (backed by a connector) proxying docker.io images; essentially the same flow I'd like to use for our internal images.

      I've pulled images through it with S3 enabled since the beginning. I'm expecting to see some data in there;

      Screenshot 2026-10-23 at 09.20.31.png

      I've tested the storage, and it seems to be happy;

      Screenshot 2026-10-23 at 09.19.59.png

      It does seem to be storing manifests, but no actual images;

      Screenshot 2026-10-23 at 09.34.38.png

      I have caching enabled on the connector;

      Screenshot 2026-10-23 at 09.36.35.png

      Inspecting ProGet itself (using the Docker image); it seems to be storing everything locally in /var/proget/packages;

      Screenshot 2026-10-23 at 09.43.57.png

      Any idea what might be going on?

      posted in Support
      C
      cyril
    • Caching ProGet feeds in S3 and/or auto-creating local repository

      Hi all,

      I'm currently in the process of migrating between ProGet instances; we're moving from a couple monolithic feeds to narrower, team-managed feeds, as well as moving from locally stored artifacts to S3-backed storage...

      Since we've got a large amount of build pipelines & deployment environments that need to be updated, we figured that we could proxy our "old" instance, using connectors + team-specific filters. This works for switching endpoints easily, without a big-bang migration. Our teams can then progressively update to push to the new instance, while old tags are pulled from the old one. This also saves us from pulling data we don't need to the new instance.

      One issue has come up however; the cached packages seem to be stored directly on disk, instead of being pushed to the (configured) S3 storage on the feed. Is this intended, or can it be reconfigured?

      A related thought, to make our migration more 'resilient' would also be to automatically create a local repository when pulling via a connector, if that could be an option. However the S3 issue still remains; we won't be able to keep all our images locally.

      Would you have any pointers on dealing with this?

      Thanks,
      Cyril

      posted in Support
      C
      cyril
    • Support for setting Debian feed distribution/components via pgutil / API

      Hello!

      We're using pgutil to programmatically manage our ProGet instance, which we're planning to use to proxy APT/Debian repositories. Creating a feed+connector works via the UI, but I can't seem to find a way to set Debian feed-specific properties, such as components & distribution, via pgutil or the API. Is this possible?

      We're setting up GitOps-style management of our ProGet instance; working great for other feed types, so would be fantastic to be able to do so with Debian connectors too!

      Best,
      Cyril

      posted in Support
      C
      cyril
    • ProGet license restrictions in CI

      Hi Inedo team; I'd like to confirm the restrictions for ProGet licenses, and how they apply for automated testing. I have already read the documentation at https://docs.inedo.com/docs/proget/administration/license

      We're currently running a Basic licensed instance for a production environment, and a Free licensed instance for testing configuration and automations. These are automations where ProGet itself is the target.

      We'd like to implement some testing via CI/CD pipelines, which would involve spinning up a ProGet instance and programmatically configuring it. No technical blockers, however there is a concern on whether this would be acceptable within the Free license terms. This would technically be a single instance, though short-lived and frequently recreated. Does this fit within the usage restrictions?

      posted in Support
      C
      cyril
    • 1 / 1