Use on-prem S3 (ceph/rgw) as feed storage

[karsten.meichsner_9039]

When I configure our on-prm S3 as feed storage , I got these error message:
System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid because of errors in the certificate chain: UntrustedRoot

We have a self-signed CA, so this error is usual, but how can we trust our own root?

[atripp]

Hi @karsten-meichsner_9039 ,

This error is happening at the SSL-level, which is managed by the operating system (platform). There's nothing you can configure in ProGet to resolve this, so you'll need to make some changes to the operating system.

Can you tell me, are you using ProGet for Windows, or on Linux (Docker)? In either case, you'll need to trust the root certificate, but the ways to do this are a bit different.

In Windows, the easiest way is to install the certificate using the UI (Certificates-Current User > Trusted Root Certification Authorities into Certificates (Local Computer) > Trusted Root Certification Authorities). You can also verify that the installation was successful by trying to navigate to a URL in your S3 bucket.

In Linux/Docker, it's a bit trickier; in general you'll want to copy the .crt into /usr/local/share/ca-certificates and then run /usr/sbin/update-ca-certificates. There's a few ways to do this, but one common way is to build a docker image on top of our image. You can also SSH into the running container and handle this as well.

Cheers,
Alana

[karsten.meichsner_9039]

Hi @atripp,

you are compleatly right! We need our root cert in the trusted root collection of the OS. We have an k8s deployment, so i added the cert with a config map. And bingo! I've got it working... but only with a very dirty trick. At the first try I got the error message:

System.Net.Http.HttpRequestException: Name or service not known ([bucket-name].[sub-doamin].s3.[domain]:443)

This means, proget is expecting the "virtual host spelling" for S3. Our Ceph System instead is expecting a "path spelling" like this:

https://[sub-doamin].s3.[domain]:443/[bucket-name]

My first idea was to use the prefix field in ProGet as the bucket-name, but this also dit not work, beaucause a bucket-name is required, so I used our DNS sub-domain name as the bucket-name, and this works! ;-)

May I ask one more question: I enabled "Use server-side encryption" and I was wondering if this was working wihout give an encrypten-key-ID for the KMS. Which key uses ProGet for the encryption?

Thank you!
Karsten

[atripp]

Hi @karsten-meichsner_9039,

I'm glad you could solve the issue! I'm not very familiar with Ceph, but it sounds like it works with the AWS API, which we use to talk to AWS. I added a brief notes to the docs about Ceph/RGW usage, but if you have suggestion to improve, please let me know :)

As for the "Use server-side encryption" option, the checkbox sets the option on the API Request (e.g. CopyObjectRequest) to use AWS Server-side Encryption. ProGet does not encrypt the data.

Cheers,
Alana

[karsten.meichsner_9039]

Hi @atripp,

yes, in general your S3 API works well with Ceph. I can only make this suggestion to improve the setting of the access data:

The difference between the two access methods to S3 is described here: https://docs.ceph.com/en/quincy/radosgw/s3/commons/

Ceph and we prefer the first method because DNS wild cards are difficult for us. AWS uses the second method, and it's the only one that ProGet supports. It would be great for all Ceph users if proget supported the first method as well.

But for now, the weekaround is OK for us.

cheers
karsten

[atripp]

Hi @karsten-meichsner_9039 ,

We'll definitely keep that in mind; we currently use the AWS SDK, which means we don't work at the HTTP-level and can't easily control the requests.

Basically our code does this:

new CopyObjectRequest
{
    SourceBucket = this.BucketName,
    SourceKey = this.BuildPath(sourceName),
    DestinationBucket = this.BucketName,
    DestinationKey = this.BuildPath(targetName),
    CannedACL = this.CannedACL,
    ServerSideEncryptionMethod = this.EncryptionMethod,
    StorageClass = this.StorageClass
},

If you know of any way to configure the SDK to send a different request, we'd be happy to try that out!

Cheers,
Alana

[karsten.meichsner_9039]

Hi @atripp ,

I'm not very familiar with C# and .NET, but found here that the AmazonS3Config class has a forcePathStyle attribute and a ForcePathStyle method. by default this is false. I think setting this to true should work correctly with Ceph/RGW.

 public partial class AmazonS3Config : ClientConfig
    {
        ...
        private bool forcePathStyle = false;
        ...
        public bool ForcePathStyle
        {
            get { return forcePathStyle; }
            set { forcePathStyle = value; }
        }
        ...
}

Cheers
Karsten

[atripp]

Hi @karsten-meichsner_9039 ,

Oh wonderful!

If you're able/interested to try it out, then we should easily be able to add a checkbox in the UI for it and send you a patch/prerelease to try.

That's how many of the other options got added

Alana