pgutil uploads error for large assets. HTTPS uploads show “operation cancelled” whilst HTTP is fine

mmaharjan_0067

Hi team,

We’re seeing pgutil uploads of large files semi-fail over HTTPS (behind a reverse proxy), while the same uploads succeed over HTTP (direct to ProGet).

Uploading a 9.2 gb file over HTTPS, the transfer hits 100% in 3 min, then after a 2 min consolidation, pgutil reports “operation cancelled.” The asset becomes present in ProGet in both cases, but that failure prevents us from setting metadata straight after in our CI/CD pipelines.

A list of tests done:

HTTP (bypassing reverse proxy/WAF), same file completes and metadata can be set.
Zipped version of the file (approx 4 gb) also fails over HTTPS but when using --partsize=1000 succeeds for this 4 gb file.
--partsize=2000,1000 and 50 on the 9.2 gb file fails.
Direct curl PUT of the same 9.2 gb file to HTTPS succeeds.
8 gb dummy file also fails over HTTPS.
4 gb ISO file uploads fine to HTTPS.

This suggest there might be something specific about how pgutil handles large, long requests along with the step that consolidates the asset. Is there recommended guidance for pgutil with very large assets.

This is pgutil v2.1.13 running the proget server on docker container using compose.

Logs show that we aren’t getting the final 200 response when it fails:

Request starting HTTP/1.1 POST http://<proget-server>/endpoints/test/content/testfile?id=7fbe72d4b20b4492a146e93cd26be868&multipart=complete - - 0
Running Feed Replication...
Feed Replication completed.
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/1.1 POST http://<proget-server>/endpoints/test/content/testfile?id=7fbe72d4b20b4492a146e93cd26be868&multipart=complete - 200 0 - 8064.5135ms

dean-houston

Hi @mmaharjan_0067 ,

It sounds like you're on the right track with researching this; your reverse proxy is definitely "breaking things" somehow.

Based on what you wrote, it sounds like your reverse proxy is terminating the request because there's no output from the server after a while. The "no output" is expected, since assembling the upload takes quite some time, and that's likely where the "operation cancelled" would be coming from.

I would look there and see if you can adjust timeouts. As for pgutil, here's the code used to perform the multi-part upload:
https://github.com/Inedo/pgutil/blob/thousand/Inedo.ProGet/AssetDirectories/AssetDirectoryClient.cs#L197

-- Dean