Docker image attempts chown on NFS

[tyler_5201]

When the pod starts, there's this error message
Initializing embedded database...
chown: changing ownership of '/var/proget/database': Operation not permitted

When inside the pod, we can see that permissions are properly set, as also verified on the NAS and via the K8 config.

ls -la /var/proget
total 0
drwxr-xr-x. 7 root root 82 Aug 22 02:52 .
drwxr-xr-x. 1 root root 20 Aug 22 02:52 ..
drwxrwxrwx. 1 1024 users 0 Aug 22 02:49 backups
drwxrwxrwx. 1 1024 users 0 Aug 22 02:54 database
drwxr-xr-x. 2 root root 6 Aug 22 02:52 extensions
drwxrwxrwx. 1 1024 users 0 Aug 22 02:49 packages
drwxr-xr-x. 2 root root 6 Aug 22 02:52 ssl

apiVersion: apps/v1
kind: Deployment
metadata:
name: proget
namespace: proget
spec:
replicas: 1
selector:
matchLabels:
app: proget
template:
metadata:
labels:
app: proget
spec:
securityContext:
runAsUser: 1024
runAsGroup: 100
runAsNonRoot: false
containers:
- name: proget
image: proget.inedo.com/productimages/inedo/proget:latest
ports:
- containerPort: 8624
name: http
volumeMounts:
- name: proget-data
mountPath: /var/proget/database
- name: proget-package
mountPath: /var/proget/packages
- name: proget-backup
mountPath: /var/proget/backups
volumes:
- name: proget-data
persistentVolumeClaim:
claimName: proget-data
- name: proget-package
persistentVolumeClaim:
claimName: proget-package
- name: proget-backup
persistentVolumeClaim:
claimName: proget-backup

This appears to be something in the setup of ProGet that is trying to perform a chown, which is not allowable on the NFS share.

[atripp]

Hi @tyler_5201

I'm not really familiar with all this topic, but why would you want to put the database file on an NFS? The PostgreSQL docs on clustering seem to warn against this.

That said, we added the chown command during early testing because PostgreSQL was failing with something like this:

The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The only way to get to work was:

1. Log into the pod via: $kubectl exec -n -i -t -- bash -il
2. Change the directory to: $cd /var/opt
3. Get the UID of postgres-User: $id -u postgres
4. Recursively change owner of the folder database: chown -R 101:101 database/

Hence, why we added the chown.

I'm not sure the best approach here -- we could try/catch, but I think the postgres user still needs to own the files?

Cheers,
Alana

[tyler_5201]

I chose NFS over iSCSI and SMB because it is more standard them then, but also easier to manage than iSCSI. The shares were originally setup without squashing, but then it became a mess to keep permissions on the NAS to match the pods. It's been far easier telling a few pods to use the UID/GID rather than to manage both the deployments & the NAS permissions.

You can't chown within the containers, without using sudo which is why this fails. I can run chown on the NAS to get the permissions to match (I'll have to test tomorrow, DHCP is being a pain on the new servers being added).

For your last question, I would have assumed the UID/GID configured would have taken over for the postgres user, and would make the chown irrelevant at that point. I'm actually running into a similar problem with the AWX helm chart because they forcefully use chmod 755 and chown 1000 without allowing any way to bypass it. The permissions are already set, so there's no need to change them.

[atripp]

Hi @tyler_5201

Thanks for the additional info; so it's not that this is an actual "network drive" just (i.e. far from the server), it just happens to be how the configuration works.

I don't think we can/should try to detect the owner; do you if chown will fail if the user is already postgres? If so, then we can wrap this with a try/catch and throw a more helpful error.

Otherwise, we can try/catch and give a warning. But we want to make sure the user can see it when the database fails to load.

Thanks,
Alana

[tyler_5201]

The chown fails no matter what because it needs to be run as sudo.

I could set the permissions on the folder, but I'd have to create a new network share because I'm mapping all Kubernetes connections to 1024:100 to keep the pods consistent. For the most part, this usually fixes any access issues to the folders, if a pod has issues.

      securityContext:
        runAsUser: 1024
        runAsGroup: 100
        runAsNonRoot: false

[atripp]

Hi @tyler_5201 ,

We'll address this via PG-3098 in an upcoming maintenance release; basically we will add a permission check before chown on Docker/Linux for PostgreSQL. If the permissions are there, then chown won't be run.

Thanks,
Alana