nvidia-container-cli not discoverable by gVisor/runsc when toolkit.installDir is non-default

[Vincinator]

Describe the bug

When toolkit.installDir is set to a non-standard location (required for OSes with a read-only /usr), nvidia-container-cli is installed to a path like /opt/nvidia/toolkit/nvidia-container-cli. This path is not on the system or containerd $PATH.

gVisor's runsc uses exec.LookPath("nvidia-container-cli") to locate the binary (source — see nvProxyPreGoferHostSetup), which fails with:

failed to locate nvidia-container-cli in PATH:
exec: "nvidia-container-cli": executable file not found in $PATH

This works with runc because nvidia-container-runtime resolves the binary path from its own config TOML. gVisor intentionally does not parse that config (source) and relies on standard PATH lookup instead.

To Reproduce

1. GPU Operator installs toolkit DaemonSet with toolkit.installDir set to a non-default path (e.g. /opt/nvidia/toolkit/).
2. gVisor/runsc is configured as an alternative runtime with nvproxy: true.
3. A GPU pod with runtimeClassName: gvisor fails to start because runsc calls exec.LookPath("nvidia-container-cli") during container creation, and the binary is not on the inherited PATH.

Expected behavior

nvidia-container-cli should be discoverable via standard PATH lookup after toolkit installation, regardless of installDir. Note that the motivation for using a non-default installDir is that /usr is read-only on these OSes, so symlinking into /usr/local/bin is not a viable solution.

Workaround

Extend containerd's PATH via a systemd drop-in:

# /etc/systemd/system/containerd.service.d/nvidia-toolkit.conf
[Service]
Environment=PATH=/opt/nvidia/toolkit:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

Followed by systemctl daemon-reload && systemctl restart containerd.

Environment:

• nvidia-container-toolkit version: v1.17.4 (also reproduced with v1.19.1)
• NVIDIA Driver Version: 590.48.01 (open kernel modules)
• Host OS: Garden Linux 2150.4 (read-only /usr)
• Container Runtime Version: containerd 2.0.x
• CPU Architecture: x86_64
• GPU Model(s): T4
• Kubernetes: 1.31
• NVIDIA GPU Operator version: 24.9.2
• gVisor/runsc version: release-20260511.0

[dims]

What's your suggestion here @Vincinator ?

[Vincinator]

When toolkit.installDir is set to a non-default path, the toolkit should ensure installDir is on containerd's PATH, for example by installing a systemd drop-in:

# /etc/systemd/system/containerd.service.d/nvidia-toolkit-path.conf
[Service]
Environment="PATH=/opt/nvidia/toolkit:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

This way, any binary that containerd spawns (including runsc) inherits the correct PATH and can locate nvidia-container-cli via standard lookup.

This requires a daemon-reload followed by a service restart to take effect. The toolkit already supports restarting the runtime service via systemd restart mode, so adding a daemon-reload before that restart seems like a natural fit.

On the UX side, this could be gated behind a configuration option so it only takes effect when needed. But that's an implementation detail.