Hi,
I've solved the problem now.

We have reverse proxy and the max request body size value was not set.
I think that resulted in the proxy only allowing very small sized packages through, causing the slow network.

We've setup a new Proget server on docker for using an internal Chocolatey repository.
Uploading and downloading .nupkg files are very slow, around 4mbit, even on local LAN.

I'm not sure where to even start troubleshooting.
I can't say I experience the web interface slow.
Other docker containers in the same swarm has no issues.