error pushing large docker images when using Azure blob storage

brett.polivka

I'm attempting to push some Windows-based images, which contain some very large, multi-GB layers. Theses pushes never succeed, as I keep getting the following errors in the server logs:

A 500 error occurred in docker: The request body is too large and exceeds the maximum permissible limit.

Where "docker" is the name of my feed.

I'm using Azure blob store as the storage backend, and from what I can tell, this is where the error is actually coming from. My guess is that the issue stems from the 100MB block size limit imposed by Azure for block blobs.

https://docs.microsoft.com/en-us/rest/api/storageservices/understanding-block-blobs--append-blobs--and-page-blobs

NanciCalo

@brett-polivka I know that there were some recent changes to these extensions; what version are you using of ProGet and the Azure extension (Admin > Extensins)?

brett.polivka

Proget version: 5.3.23
Azure extension version: 1.11.0

NanciCalo

@brett-polivka thanks

So, this may have been a regression due to a recent change to the Azure extension. Can you rollback to Azure 1.9.0 by manually installing it?

brett.polivka

Any advice on how to do that when using the docker version?

gdivis

@brett-polivka It looks like the underlying problem is likely that 100MB limit. When we published the Azure extension, we'd mistakenly believed the maximum block size to be 4000 MB, but because we're using the v11 client library to push to azure, it's limited to the older 100mb limit. We can't upgrade without dropping support for .NET 452 on windows, which we aren't prepared to do yet, so we'll work around using another means.

We're on it; hopefully will have an update included for tomorrow's release.

gdivis

@brett-polivka So one more update. Implementing the workaround was easier than expected. This should now be resolved in v1.11.1 of the Azure extension, which you install from the Admin->Extensions page. You may need to restart the Docker container after installing.

brett.polivka

Looks good!

I had to create a volume mount for /var/proget/extensions to get the extension upgrade to persist, but it seems to fix the problem.

Thanks!