<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[Bug: ProGet caches invalid Docker layer blob, keeps serving it]]></title><description><![CDATA[<p dir="auto">An issue I unfortunately have been running into recently; we're using ProGet with connectors to other registries. For an unknown reason (I suspect some network issues upstream), large-ish images (e.g., 400mb) that are pulled by ProGet through its connector get dropped. Likely due to the download just taking longer, therefore more opportunity for the connection to drop. Timeouts and related settings are within reasonable amounts.</p>
<p dir="auto">In any case, the result is that ProGet shows that image as available and cached, but when pulled by a client, it fails due to size mismatch:</p>
<pre><code>docker pull proget.myorg.com/...
...
e28c9cbe4ea2: Download complete
short read: expected 64432621 bytes but got 38190973: unexpected EOF
</code></pre>
<p dir="auto">The only way I've found to fix this (manually) is to stop clients from requesting the multiple images that might share that layer, then delete the image, then delete the metadata cache, and finally run the Docker/Feed cleanup executions. Feed integrity shows green, and running the cleanup without deletion also had no effect. Having to stop the downstream clients was also a concern: I was otherwise racing before they requested the image again &amp; repopulated the (invalid) cache.</p>
<p dir="auto">It essentially seems like ProGet is keeping partial/corrupted blobs, and then serving them as though they were correct, with no check and no mechanism to resolve itself. Is there anything I can do here?</p>
<p dir="auto">(Additional context: running 2026.14 Linux image)</p>
]]></description><link>https://forums.inedo.com/topic/5868/bug-proget-caches-invalid-docker-layer-blob-keeps-serving-it</link><generator>RSS for Node</generator><lastBuildDate>Fri, 06 Nov 2026 15:55:39 GMT</lastBuildDate><atom:link href="https://forums.inedo.com/topic/5868.rss" rel="self" type="application/rss+xml"/><pubDate>Thu, 05 Nov 2026 18:50:58 GMT</pubDate><ttl>60</ttl><item><title><![CDATA[Reply to Bug: ProGet caches invalid Docker layer blob, keeps serving it on Fri, 06 Nov 2026 10:37:55 GMT]]></title><description><![CDATA[<p dir="auto"><a class="plugin-mentions-user plugin-mentions-a" href="/user/atripp" aria-label="Profile: atripp">@<bdi>atripp</bdi></a> thanks, I'll investigate some more in that case. In case it might trigger other thoughts: I forgot to mention we're using cloud storage (S3, on premise so still very high throughput), with common blob storage disabled.</p>
]]></description><link>https://forums.inedo.com/post/20187</link><guid isPermaLink="true">https://forums.inedo.com/post/20187</guid><dc:creator><![CDATA[cyril]]></dc:creator><pubDate>Fri, 06 Nov 2026 10:37:55 GMT</pubDate></item><item><title><![CDATA[Reply to Bug: ProGet caches invalid Docker layer blob, keeps serving it on Fri, 06 Nov 2026 07:25:13 GMT]]></title><description><![CDATA[<p dir="auto">Hi <a class="plugin-mentions-user plugin-mentions-a" href="/user/cyril" aria-label="Profile: cyril">@<bdi>cyril</bdi></a> ,</p>
<p dir="auto">It's hard to really say but it could be a client/caching issue.</p>
<p dir="auto">I can describe ProGet's behavior:</p>
<ul>
<li>when adding a blob to a feed, ProGet first requests an exclusive write lock on a file, writes the entirety of the stream to that file while hashing it, confirms the hash, then writes a database entry indicating the successful file write and size</li>
<li>when downloading a blob from a connector, a race condition may occur when adding the cached file to the feed; this is ignored, and the remote stream is returned</li>
<li>when performing a Docker feed scan, ProGet will recompute the hash of each blob and report on issues</li>
<li>when performing a reindex (with deletion), ProGet will perform another hash recompute and delete mismatched blobs</li>
</ul>
<p dir="auto">That said, aside from OS-level interference -- which happens A LOT due to endpoint monitoring tools -- we're not aware of any conditions in which a blob is truncated or fails to write. I can't imagine how the integrity check would fail to detect a truncated blob.</p>
<p dir="auto">The next time you see an error like this, I would check the content on ProGet server itself. For example, when you see this:</p>
<pre><code>e28c9cbe4ea2: Download complete
short read: expected 64432621 bytes but got 38190973: unexpected EOF
</code></pre>
<p dir="auto">Track down the <code>e28c9cbe4ea2</code> blobfile in the package store. It should have a hash that matches the digest.</p>
<p dir="auto">Separately, you can test ProGet's behavior by going to the file store and manipulating a blob file. For example, rename <code>abcd1234</code> to <code>zzz_abcd1234</code> then copy some other blob and name it <code>abcd1234</code>. That will always fail a hashcheck.</p>
<p dir="auto">Thanks,<br />
Alana</p>
]]></description><link>https://forums.inedo.com/post/20185</link><guid isPermaLink="true">https://forums.inedo.com/post/20185</guid><dc:creator><![CDATA[atripp]]></dc:creator><pubDate>Fri, 06 Nov 2026 07:25:13 GMT</pubDate></item></channel></rss>