Universal Package Upload large files >~2GB fails

[Guest]

Hi,
Is there a maximum size for packages in the Universal Package Repository?

We have a large number of packages that are between 2 and 5GB in size (and cant be made smaller). I've previously updated web.config to allow for files larger than this size.

I'm using chrome so I can see the %upload progress and i notice that with files larger than 2GB when they reach roughly 2GB uploaded then the upload pauses for around 30 seconds and then seems to timeout returning to the Upload package screen.

Is there a fix or a workaround to get the files in?
NB the files are already on the server running proget to save transfer time

Product: ProGet
Version: 4.7.9

[ben]

Hello Marcus,

There was a bug in our HTTP library that caused uploads larger than 2GB to fail or be truncated. It will be fixed for the next version of ProGet, 4.7.10.

[Guest]

I've just downloaded 4.7.10 and the problem still exists using push.
and if you try and upload from the web page you get

[HttpException (0x80004005): Maximum request length exceeded.]
System.Web.HttpRequest.GetEntireRawContent() +1105
System.Web.HttpRequest.GetMultipartContent() +83
System.Web.HttpRequest.FillInFormCollection() +229
System.Web.HttpRequest.EnsureForm() +88
System.Web.HttpRequest.get_Form() +16
Inedo.Web.PageFree.SimplePageBase.ProcessControlTree() +221
Inedo.Web.PageFree.<ProcessRequestAsync>d__48.MoveNext() +366
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) +14139120
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) +62
System.Web.TaskAsyncHelper.EndTask(IAsyncResult ar) +64
System.Web.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +627
System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +146

[ben]

Hello Marcus,

In IIS Manager, select your ProGet site and open the configuration editor.

In the section named system.webServer/security/requestFiltering, expand requestLimits. Set maxAllowedContentLength to at least the maximum size of uploads you expect in bytes.

In the section named system.web/httpRuntime, set maxRequestLength to at least the maximum size of uploads you expect in kilobytes.

[Guest]

We had this problem so set the values to be max values, using int.maxvalue but now we are experiencing a different error.

System.ArgumentOutOfRangeException: Non-negative number required.
Parameter name: count
at System.IO.FileStream.WriteAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
at System.IO.Stream.<CopyToAsyncInternal>d__2.MoveNext()

any ideas? could the image be corrupt?

[Guest]

Just an FYI. ..

We're seeing a lot of unfortunately limitations within the general infrastructure components (IIS, ASP.NET hosting, etc) as files get larger and larger.

So, we're going to be doing a sort of "chunking API" with the release of ProGet 4.8, and then migrate it to universal packages.

[Guest]

Thanks for the feedback Alana

So is this a known issue?

When can we expect 4.8 to be available?

Thanks

[Guest]

HTTP/S is simply not designed for such large requests, so you'll see some problems with it like this.

4.8 is in beta now; we will have the chunking api within a week or so.

[Guest]

Hi Alana

can you confirm the chunking api was included in the 4.8.2 release or earlier?

i upgraded last night to 4.8.2 from 4.7.14 but still suffering the same fate with large docker images

Thanks
Geoff

[Guest]

Hi Proget Support

Can you confirm the above please.

We would like to try and use the docker feed, but as part of that we will need to store some large images.

Thanks

[Guest]

We've currently only implemented the API in Asset Directories; see multipart-upload API; we'd definitely welcome your feedback on it.

It seems like it would work for Universal Packages in a similar manner, but we want to test it out for assets first.

[john.alexander_6448]

Did the multi-part API get implemented for universal packages? This is still an issue.