[faganihajizada]

Problem

CDI spec generation mounts IPC sockets (nvidia-persistenced, nvidia-fabricmanager ...) with the ro (read-only) mount option. This breaks nested container runtimes like enroot/pyxis that need to bind-mount these sockets into containers.

When we try to run slurm job with enroot/pyxis on K8s:

pyxis: imported docker image: nvcr.io/nvidia/cuda:12.4.0-base-ubuntu22.04
error: pyxis: container start failed with error code: 1
error: pyxis: printing enroot log file:
error: pyxis:     nvidia-container-cli: mount error: mount operation failed: 
                  /tmp/enroot/data/user-0/pyxis_16.0/run/nvidia-persistenced/socket: operation not permitted
error: pyxis:     [ERROR] /etc/enroot/hooks.d/98-nvidia.sh exited with return code 1

Root Cause

The ro option is inherited from the default mount options in mounts.go, but IPC sockets should not be read-only. This is inconsistent with libnvidia-container which does not use MS_RDONLY for IPC mounts (reference).

Fix

Define IPC-specific mount options in ipc.go that exclude ro, matching libnvidia-container behavior:

var ipcMountOptions = []string{
    "nosuid",
    "nodev", 
    "rbind",
    "rprivate",
    "noexec",
}

Testing

• Unit test updated and passing
• Manually verified on AWS EKS with host-installed NVIDIA drivers
• Confirmed enroot container starts successfully after fix

Additional Context

I tested two AWS EKS clusters with identical GPU Operator versions:

• When GPU Operator manages drivers, nvidia-persistenced runs inside the driver container. The socket is part of the overlay filesystem and it works fine.
• When using host-installed drivers (e.g., AWS AMI), CDI discovers the host socket and mounts it as tmpfs (ro)

This is critical to support SlurmonK8s (https://github.com/SlinkyProject/slurm-operator) with enroot/pyxis on K8s.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[elezar]

Looks good. Thanks.

[elezar]

/cherry-pick release-1.18

[coveralls]

Pull Request Test Coverage Report for Build 21167707185

Details

• 1 of 1 (100.0%) changed or added relevant line in 1 file are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage remained the same at 36.837%

---

Totals
Change from base Build 21132355090:	0.0%
Covered Lines:	5257
Relevant Lines:	14271

---

💛 - Coveralls

[elezar]

/ok-to-test fb15d14

[ArangoGutierrez]

lgtm

[github-actions]

🤖 Backport PR created for release-1.18: #1594 ✅