Max file upload

[russell_8876]

Hi, I've just ran into the 2GB upload limit...

Having a search on the forum it look as though theres not a high demand for large file distribution?

My use case is setting up ProGet to host/distribute compiled artefacts from a digital twin pipeline. The Unreal app im working with at the moment is 24gb.

Ive got the linux docker container running. Im hoping its a simple case of changing a web server configuration parameter?

Thanks in advance

[stevedennis]

Hi @russell_8876,

ProGet itself does not have an upload limitation, so it's likely something else like your reverse proxy server, etc. Typically such large requests are blocked/prevented by middleware.

That said, HTTP is not a reliable protocol and should never be used for such large requests. You will run into a lot of problems trying to upload 24GB files in a single request. You'll need a new approach.

The easiest solution is to use drop paths, and then a file transfer protocol that is designed for large/reliable file transfers (most are).

Another solution is to use "upload chunking", which only asset directories support; pgutil should handle the chunking/uploading for you. If you want to use packages, you can can then import that uploaded asset into a universal package feed:
https://docs.inedo.com/docs/proget/upack/proget-api-universalfeed/proget-api-universalfeed-import

-- Dean

[russell_8876]

Hi,

Thank you Dean for the detailed response, much appreciated.

I'm still settings things up, so just working on the latest linux docker container (proget.inedo.com/productimages/inedo/proget:latest) so no reverse proxy involved just yet.

For context, I'll be setting up a build server soon, it will be this that i want to programatically add to Proget.

Im using pgutil to upload to the assets feed, so great to hear http chunking is happening behind the scenes. Yes, no 24gb files are being uploaded via the web UI :-). However, its this im using to upload a test project thats 24GB to the docker container and i hit the 2gb limit, it just bails out at 2GB.)

Im not that saavy with windows web servers, i've had a poke around and found this config
/usr/local/proget/service/web.config

and this line

<requestLimits maxAllowedContentLength="4294967295" maxQueryString="1048576" maxUrl="1048576" />

Which is confusing as im getting stopped at 2GB, where as this is set to 4GB... not quite sure what this is. I increased it, but had no effect.

Attached is a screen shot of the response from the server i get when i upload a 25gb files, getting blocked at 2GB

Attached is a screenshot of the docker logs, highlight at the point where the upload fails

It would be good to work out how to get pgutil to be able to upload a 2gb+ asset directly, as this is the most simple. To side step this issue, I have set up a drop path and this gets the file on the stream, but stores at the root and doesn't seem to be a way of saving directly to the sub directory? I have a latest/latest_archive/WIP folders that the build server will be saving to.

In the WebUI there is a move function, but I'd like to achieve this without manual intervention. Ive not been able to work out how to use the Web API to achieve a move, any help here is much appreciated.

[steviecoaster]

Are we sure this is a ProGet problem and not a Docker problem? Assuming the asset is outside the container and you're trying to upload it into ProGet inside the container, you may be hitting a setting inside Docker.

[dean-houston]

Hi @russell_8876,

Thanks for all the additional details!

Actually, this is definitely a bug in pgutil, since it's sending a negative offset to ProGet, and then ProGet is responding with a 400 error ("Missing or invalid "offset" parameter.)

Asset uploaf capability is just a few weeks old, and I'm not sure if we tested with such a large file. Anyway, we'll get it looked at and let you know once fixed; just a small tweak needed here I think:

https://github.com/Inedo/pgutil/blob/thousand/Inedo.ProGet/AssetDirectories/AssetDirectoryClient.cs#L240

-- Dean

[gdivis]

Thanks @russell_8876! I've gone ahead and released pgutil 1.1.6 with a fix for this. Give it a try and let us know if it's still not working.

-Greg