Help mounting nvoptix.bin in containers

[emaincourt]

Hi,

After trying to upgrade our Bottlerocket AMIs, we realized OptiX was broken. The reason behind that is that, I supposed in recent OptiX versions, nvoptix.bin is required to load the denoiser weights according to the error logs.

I updated packages/kmod-6.1-nvidia/kmod-6.1-nvidia.spec to install the file when building the package. See the commit here. I'm pretty new to Bottlerocket so I might actually do it wrong. Anyway the file is properly available when I boot the host with my custom AMI at path /usr/share/nvidia/nvoptix.bin which is the proper path for the nvidia-container-toolkit to load it.

However, when I start a container with the right environment variables (eg. NVIDIA_DRIVER_CAPABILITIES=compute,utility,graphics,display), the file does not get mounted into the container. I tried the same thing with an AL2023 AMI which works as expected. nvoptix.bin is located at the same path, and properly gets mounted into the container.

The nvidia-container-runtime configuration looks quite different on the AL2023 AMIs and the Bottlerocket ones:

# Bottlerocket
accept-nvidia-visible-devices-as-volume-mounts = true
accept-nvidia-visible-devices-envvar-when-unprivileged = false

[nvidia-container-cli]
root = "/"
path = "/usr/bin/nvidia-container-cli"
environment = []
ldconfig = "@/sbin/ldconfig"

# AL2023
#accept-nvidia-visible-devices-as-volume-mounts = false
#accept-nvidia-visible-devices-envvar-when-unprivileged = true
disable-require = false
supported-driver-capabilities = "compat32,compute,display,graphics,ngx,utility,video"
#swarm-resource = "DOCKER_RESOURCE_GPU"

[nvidia-container-cli]
#debug = "/var/log/nvidia-container-toolkit.log"
environment = []
#ldcache = "/etc/ld.so.cache"
ldconfig = "@/sbin/ldconfig"
load-kmods = true
#no-cgroups = false
#path = "/usr/bin/nvidia-container-cli"
#root = "/run/nvidia/driver"
#user = "root:video"

[nvidia-container-runtime]
#debug = "/var/log/nvidia-container-runtime.log"
log-level = "info"
mode = "auto"
runtimes = ["docker-runc", "runc", "crun"]

[nvidia-container-runtime.modes]

[nvidia-container-runtime.modes.cdi]
annotation-prefixes = ["cdi.k8s.io/"]
default-kind = "nvidia.com/gpu"
spec-dirs = ["/etc/cdi", "/var/run/cdi"]

[nvidia-container-runtime.modes.csv]
mount-spec-path = "/etc/nvidia-container-runtime/host-files-for-container.d"

[nvidia-container-runtime-hook]
path = "nvidia-container-runtime-hook"
skip-mode-detection = false

[nvidia-ctk]
path = "nvidia-ctk"

However I do not feel like this would be the issue. If I'm right, the nvidia-container-runtime-hook binary is in charge of mounting the files and the configuration file from AL2023 does not seem to set any parameter that would make it different.

There is a specific configuration property that can be set to enable debugging logs for the nvidia container runtime:

debug = "/var/log/nvidia-container-runtime.log"

Enabling that setting actually logs the files that are being mounted on AL2023. On Bottlerocket, it does nothing. I'm wondering maybe the binary is not allowed to write that path but I might be wrong.

Basically my questions would be:

• Do I install the files properly to get the nvidia runtime to access it and be able to mount nvoptix.bin ?
• Is there any way to enable the debug logging properly so I could get some hints about the underlying issues ?

Thanks in advance for any help

[vyaghras]

Hi @emaincourt,

• You mentioned that you are creating custom AMI, Can you tell which version of Bottlerocket/ Bottlerocket-core-kit you are creating fork from ?
• Is it possible for you to share the linked commit so that we can access.

[emaincourt]

Hi @vyaghras,

Thanks for your answer. I tried it using the latest release back in October, just to make sure I was using a stable base. Also I'm deeply sorry I did not realize the repository was private ! Now fixed.

[arnaldo2792]

Hello @emaincourt, the architecture that we followed in Bottlerocket to mount NVIDIA libraries and binaries from the host into the containers is now consider "legacy", and hence the difference between the configurations for Bottlerocket and AL2023.

In the "legacy" stack, libnvidia-container was in charge of finding and mounting the libraries and binaries. However, that changed with the move to CDI and the new nvidia-container-toolkit support. This is why you don't see nvoptix.bin mounted in the containers, because the nvidia-container-toolkit is now in charge of mounting some binaries when CDI is used.

Once we migrate to our stack to the new NVIDIA Container Toolkit CDI support, the extra binaries that are "missing" in the containers should be available. We are still planning when the migration will occur, but contributions are welcome!. A potential workaround you can try is to force the mount in your pods through normal Kubernetes directives.

[emaincourt]

Hi @arnaldo2792,

Thank you for taking the time to investigate my issue. It makes a lot more sense now ! I could potentially help with that, do you have any spec or documentation regarding the "legacy" vs new architecture ?