Like stated in the title, are there deduplication capabilities within ProGet? As an organization, we have an extremely large deployment that we are looking to fully migrate to ProGet. Currently, our package managers we are running stores around 15 TB of packages across a variety of different package feeds including Debian, Conan, maven, npm and docker. Using deduplication, we are around 5 TiB of storage usage. (the majority of it is in Debian and maven, but there are quite a few in the other feed types as well)

A question along the same lines. Where would we start seeing a degradation of proget by the number of packages? We currently maintain around 5 million packages in our current solution. Would there be any performance degradation from that number of packages within ProGet?

Thanks in advance!

Thank you so much for that information. I ended up putting in this PR with the changes. This change should allow for full support of most (if not all) other S3 Compatible services like Backblaze & MinIO. I did test this specifically with MinIO & AWS with no issues presenting themselves.

If there needs to be anything else done from my end to perform testing, I would be more than happy to do that to get this over the finish line.

As we are needing to run S3 on Premise, we elected to go with MinIO
So after testing, it looks like there are compatibility issues with the AWS SDK. The issue being the default connector using the AWS SDK is using what is called virtual style rather than path style. In this case, I believe a change would be needed to support path style URL's. eg. with aws, you could use either <bucket name>.s3.amazonaws.com or s3.amazonaws.com/bucket-name. For reference from AWS, here is their documentation on the difference between Virtual Hosted & Path Style Requests.
In the situation of how the SDK works by default, it is using virtual path style which would not work with the implementation style that we have used with MinIO.

From what I can tell, the modification to the AWS SDK should be relatively simple to implement as it is adding another option within the web interface, as well as adding a line in the AmazonS3Config class to "ForcePathStyle" when selected. If it would be easier, I can draft up a PR that can implement those changes, however my C# is a bit rusty.

I will say that by implementing this, it should support the rest of the S3 Compatible providers others have looked at.

I've been looking into ProGet with my company and wanted to see if there was support for MinIO? I know there is AWS S3 storage support, but it looks like the only way is using virtual path support, whereas MinIO requires forcing path style.