Welcome to the Inedo Forums! Check out the Forums Guide for help getting started.

If you are experiencing any issues with the forum software, please visit the Contact Form on our website and let us know!

Docker image attempts chown on NFS



  • When the pod starts, there's this error message
    Initializing embedded database...
    chown: changing ownership of '/var/proget/database': Operation not permitted

    When inside the pod, we can see that permissions are properly set, as also verified on the NAS and via the K8 config.

    ls -la /var/proget
    total 0
    drwxr-xr-x. 7 root root 82 Aug 22 02:52 .
    drwxr-xr-x. 1 root root 20 Aug 22 02:52 ..
    drwxrwxrwx. 1 1024 users 0 Aug 22 02:49 backups
    drwxrwxrwx. 1 1024 users 0 Aug 22 02:54 database
    drwxr-xr-x. 2 root root 6 Aug 22 02:52 extensions
    drwxrwxrwx. 1 1024 users 0 Aug 22 02:49 packages
    drwxr-xr-x. 2 root root 6 Aug 22 02:52 ssl
    
    apiVersion: apps/v1
    kind: Deployment
    metadata:
    name: proget
    namespace: proget
    spec:
    replicas: 1
    selector:
    matchLabels:
    app: proget
    template:
    metadata:
    labels:
    app: proget
    spec:
    securityContext:
    runAsUser: 1024
    runAsGroup: 100
    runAsNonRoot: false
    containers:
    - name: proget
    image: proget.inedo.com/productimages/inedo/proget:latest
    ports:
    - containerPort: 8624
    name: http
    volumeMounts:
    - name: proget-data
    mountPath: /var/proget/database
    - name: proget-package
    mountPath: /var/proget/packages
    - name: proget-backup
    mountPath: /var/proget/backups
    volumes:
    - name: proget-data
    persistentVolumeClaim:
    claimName: proget-data
    - name: proget-package
    persistentVolumeClaim:
    claimName: proget-package
    - name: proget-backup
    persistentVolumeClaim:
    claimName: proget-backup
    

    This appears to be something in the setup of ProGet that is trying to perform a chown, which is not allowable on the NFS share.


  • inedo-engineer

    Hi @tyler_5201

    I'm not really familiar with all this topic, but why would you want to put the database file on an NFS? The PostgreSQL docs on clustering seem to warn against this.

    That said, we added the chown command during early testing because PostgreSQL was failing with something like this:

    The files belonging to this database system will be owned by user "postgres".
    This user must also own the server process.
    

    The only way to get to work was:

    1. Log into the pod via: $kubectl exec -n -i -t -- bash -il
    2. Change the directory to: $cd /var/opt
    3. Get the UID of postgres-User: $id -u postgres
    4. Recursively change owner of the folder database: chown -R 101:101 database/

    Hence, why we added the chown.

    I'm not sure the best approach here -- we could try/catch, but I think the postgres user still needs to own the files?

    Cheers,
    Alana



  • I chose NFS over iSCSI and SMB because it is more standard them then, but also easier to manage than iSCSI. The shares were originally setup without squashing, but then it became a mess to keep permissions on the NAS to match the pods. It's been far easier telling a few pods to use the UID/GID rather than to manage both the deployments & the NAS permissions.

    You can't chown within the containers, without using sudo which is why this fails. I can run chown on the NAS to get the permissions to match (I'll have to test tomorrow, DHCP is being a pain on the new servers being added).

    For your last question, I would have assumed the UID/GID configured would have taken over for the postgres user, and would make the chown irrelevant at that point. I'm actually running into a similar problem with the AWX helm chart because they forcefully use chmod 755 and chown 1000 without allowing any way to bypass it. The permissions are already set, so there's no need to change them.


  • inedo-engineer

    Hi @tyler_5201

    Thanks for the additional info; so it's not that this is an actual "network drive" just (i.e. far from the server), it just happens to be how the configuration works.

    I don't think we can/should try to detect the owner; do you if chown will fail if the user is already postgres? If so, then we can wrap this with a try/catch and throw a more helpful error.

    Otherwise, we can try/catch and give a warning. But we want to make sure the user can see it when the database fails to load.

    Thanks,
    Alana



  • The chown fails no matter what because it needs to be run as sudo.

    I could set the permissions on the folder, but I'd have to create a new network share because I'm mapping all Kubernetes connections to 1024:100 to keep the pods consistent. For the most part, this usually fixes any access issues to the folders, if a pod has issues.

          securityContext:
            runAsUser: 1024
            runAsGroup: 100
            runAsNonRoot: false
    

  • inedo-engineer

    Hi @tyler_5201 ,

    We'll address this via PG-3098 in an upcoming maintenance release; basically we will add a permission check before chown on Docker/Linux for PostgreSQL. If the permissions are there, then chown won't be run.

    Thanks,
    Alana


Log in to reply
 

Inedo Website HomeSupport HomeCode of ConductForums GuideDocumentation