Not able to upload .spd files to proget assets

parthu.reddy

Hi Team

We're using proget 2024.37 (Build 4) and trying to upload .spd files to an assets, and we're getting error like below.

We've checked with our firewall team and they mentioned this.

Pls suggest on this.

Thanks
Parthu

stevedennis

Hi @parthu-reddy ,

There's nothing in ProGet that would cause a file name ending with a .spd fail. Just to double check that it's not related to the file name, I would rename it and see if it works.

The main thing that's jumping out to me is that there are large files (3.1GB). Those are uploaded using a chunked process, orchestrated by the browser. I would keep "playing" with this and see if you can identify any patterns.

The message from your firewall team is saying that they are seeing RST messages from the client and server. That's unusual, and means the connection is forcibly closed. ProGet does not operate at the network level and there is nothing that can cause ProGet to issue an RST command, which means it's most certainly network related.

I think you're on the right track with investigating at the network side -- but unfortunately you're going to have to find out what is issuing a RST. The firewall might not, but maybe there are other network devices that are?

Maybe something is "taking too long" and getting RST.

Thanks,
Steve

parthu.reddy

Hi @stevedennis

Can we know how we could enable verbose logging to check for any connectivity failures.

Thanks
Parthu

atripp

Hi @parthu-reddy ,

Since these are network-level errors, you would need to use a tool like Wireshark or another packet analyzer to troubleshoot these kind of connectivity failures.

Thanks,
Alana

parthu.reddy

Hi @atripp

We found that as we're running 3 clustered servers behind haproxy(load balancer) these chunked data is sent to different servers and resulting in file being used by another process error.

We've changed the load balancing to sticky and all chunks are going through single server resulting in successful upload.

I believe this has to be fixed by proget. Could you look into it.

Thanks
Parthu

stevedennis

Hi @parthu-reddy,

Thanks for discovering/confirming that; unfortunately we're not able to reproduce this issue, as the multi-part / chunked uploads already take into account multiple servers.

Chunked upload sessions are persisted in the shared database (ChunkedUploads table)
Bytes are appended to a file stored in shared store

Would you be able to dig into the request patterns a little more? I suspect there's "something" configured on the load-balancer that's "doing something weird" with these ranged requests.

The Multipart Upload API explains what's happening behind the scenes, and you may find that using pgutil assets upload is easier to troubleshoot.

Thanks,
Steve