I wanted one Fedora box that compiles CUDA code and shares its GPU with containers. The first thing that broke was the repo URL. NVIDIA publishes no fedora45 tree yet, so the usual route to the CUDA Toolkit on Fedora, a repo file built on $releasever, dies with a 404 before dnf downloads a single package. The fedora44 repo installs cleanly on Fedora 45, as long as you keep NVIDIA’s own driver packages away from the RPM Fusion driver you already run.
My notes from that build cover the whole chain: the RPM Fusion driver, the CUDA Toolkit from the fedora44 repo, a test kernel compiled with nvcc against Fedora’s GCC 16, then the NVIDIA Container Toolkit and its CDI spec feeding rootless Podman, Docker and an Ollama container. Fedora 44 uses the same commands, since fedora44 is its native tree. I ran the whole stack in October 2026 on a Fedora 45 Cloud instance on AWS with one NVIDIA A10G and SELinux enforcing, and every output below comes from that box.
How the CUDA Toolkit on Fedora fits with containers
Three repos feed this setup and only one of them owns the driver. RPM Fusion builds the kernel module with akmods, NVIDIA’s CUDA repo supplies nvcc and the libraries under /usr/local/cuda, and NVIDIA’s container repo supplies the tool that writes a CDI spec describing the GPU. Containers never touch the CUDA Toolkit at all. They get the device nodes and the driver libraries the spec lists, which is why a container can run CUDA 13 code on a host that has no toolkit installed.
In the diagram, the red box marks the failure I hit first: a $releasever repo resolves to fedora45, which does not exist. Below, each layer is listed with the repo it came from on the test box, so you know which repo to blame when one of them breaks after an update.
| Layer | Package | Repo | Version tested |
|---|---|---|---|
| Kernel module and driver | akmod-nvidia, xorg-x11-drv-nvidia-cuda | RPM Fusion nonfree | 615.78.08 |
| Compiler and libraries | cuda-toolkit | NVIDIA cuda-fedora44 | 13.4.3 (nvcc V13.4.92) |
| CDI spec generator | nvidia-container-toolkit | NVIDIA libnvidia-container | 1.20.1 |
| Container engines | podman, docker-ce | Fedora, Docker CE | 6.1.3, 29.9.2 |
| Host compiler | gcc | Fedora | 16.2.1 |
What the box needs before you start
My instance was a g5.xlarge (4 vCPU, 16 GB RAM, one A10G with 24 GB) on an 80 GB root disk. Disk is the item people underestimate, because the toolkit alone lands 8 GiB:
- Fedora 45 or 44, x86_64, with sudo
- An NVIDIA GPU that
lspcican see - At least 15 GB free on
/for the driver, toolkit and a few images - Secure Boot off, or a MOK key enrolled for akmods
Secure Boot was disabled on the AWS instance, so I did not have to sign anything. On a workstation with Secure Boot on, the akmod module will not load until its key is enrolled, and the Fedora NVIDIA driver guide walks through the mokutil part. Check that the kernel sees the card first:
sudo dnf install -y pciutils
lspci | grep -i nvidiaOn the g5 instance, the A10G showed up as a 3D controller on the PCI bus.
00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1)No line at all means a passthrough or seating problem, and nothing later in this guide will fix that. Sort the hardware out first, because a driver install on a box with no visible card still succeeds and only fails at the reboot.
Installing the NVIDIA driver from RPM Fusion
I took the driver from RPM Fusion because its akmod rebuilds the module for every new Fedora kernel, and that’s the part that bites people a month after setup. NVIDIA’s repo can supply a driver too, but mixing the two is where the trouble starts (more on that in the repo section). Update first and reboot if a new kernel lands, because akmods builds for the newest installed kernel and the check below only looks at the running one:
sudo dnf upgrade -y --refreshMy image was already current (dnf said “Nothing to do”). Then enable both RPM Fusion repos:
sudo dnf install -y https://mirrors.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm https://mirrors.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpmThen install the akmod and the CUDA userspace libraries that go with the driver. xorg-x11-drv-nvidia-cuda is what puts nvidia-smi and libcuda.so on the host, so do not skip it on a headless box:
sudo dnf install -y akmod-nvidia xorg-x11-drv-nvidia-cudaOn my instance the transaction took 1 minute 55 seconds, and it dragged in PipeWire and upower as dependencies even on the cloud image. dnf returns before the module exists, though. The akmods build kept running in the background for another 90 seconds, so I waited until modinfo answered before rebooting. The loop gives up after ten minutes, and if it does, the build log in /var/cache/akmods/nvidia/ says why:
for i in $(seq 60); do modinfo -F version nvidia 2>/dev/null && break; sleep 10; doneOnce the module was built for the running kernel, the loop printed the driver version and exited.
615.78.08Reboot so the module loads instead of the nouveau driver:
sudo rebootOnce the box is back, ask the card what it is:
nvidia-smiThe driver reported itself on the 7.2 kernel:
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 615.78.08 KMD Version: 615.78.08 CUDA UMD Version: 13.4 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA A10G Off | 00000000:00:1E.0 Off | 0 |
| 0% 23C P8 11W / 300W | 0MiB / 23028MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
...Older drivers used a different header, so read this one carefully: “KMD Version” is the kernel module and “CUDA UMD Version: 13.4” is the newest CUDA runtime this driver can serve. That 13.4 is the ceiling for any toolkit or container image you run, which is why the toolkit in the next step is also 13.4. If you read that header with the older layout in mind, the nvidia-smi command guide covers every field and the query flags I use below.
Pointing dnf at the fedora44 CUDA repo on Fedora 45
NVIDIA’s download server had trees for fedora43 and fedora44 when I checked, and nothing for fedora45. A quick probe tells you where your release stands before you add anything:
for r in fedora45 fedora44 fedora43; do echo "== $r"; curl -sI https://developer.download.nvidia.com/compute/cuda/repos/$r/x86_64/repodata/repomd.xml | head -1; doneOnly the 45 tree came back missing:
== fedora45
HTTP/2 404
== fedora44
HTTP/2 200
== fedora43
HTTP/2 200 A 404 on the 45 tree is the whole problem, and it shows up in two ways depending on how the repo was added.
Error: “Status code: 404 for …/cuda/repos/fedora45/x86_64/repodata/repomd.xml”
dnf prints this when a repo file uses fedora$releasever in its baseurl, or when someone hand-edits the 44 file to say 45. My first test repo did exactly that, and dnf gave up after four attempts (trimmed here to one):
>>> Status code: 404 for https://developer.download.nvidia.com/compute/cuda/repos/fedora45/x86_64/repodata/repomd.xml (IP: ...) - ...
>>> Usable URL not foundUse NVIDIA’s own fedora44 repo file instead, which has the release hardcoded and never expands $releasever. On Fedora 44 this is simply the native repo. On Fedora 45 the packages are built against Fedora 44, and in this lab they installed and ran without a single dependency complaint. The same 404 pattern hits other vendors too, and the Fedora 45 third-party repo fixes track which ones have caught up:
sudo dnf config-manager addrepo --from-repofile=https://developer.download.nvidia.com/compute/cuda/repos/fedora44/x86_64/cuda-fedora44.repodnf5 writes the file to /etc/yum.repos.d/cuda-fedora44.repo with the release baked into every URL:
[cuda-fedora44-x86_64]
name=cuda-fedora44-x86_64
baseurl=https://developer.download.nvidia.com/compute/cuda/repos/fedora44/x86_64
enabled=1
gpgcheck=1
gpgkey=https://developer.download.nvidia.com/compute/cuda/repos/fedora44/x86_64/73CD9B30.pubWhen NVIDIA eventually publishes fedora45, swap this file for the new one and re-apply the exclude from the next section to the new repo ID (cuda-fedora45-x86_64), because the exclude is keyed to the repo ID and won’t follow the swap. Installed packages keep working across it, since only the source of future updates moves.
Why NVIDIA’s driver packages stay out
NVIDIA’s CUDA repo isn’t toolkit-only. It carries a complete driver stack of its own (nvidia-driver, kmod-nvidia-open-dkms, nvidia-settings, nvidia-persistenced), one point release behind what RPM Fusion had shipped. Would dnf quietly swap my driver for NVIDIA’s? I asked it to install nvidia-driver with --assumeno to find out:
sudo dnf install --assumeno nvidia-driverdnf refused, because the two stacks declare conflicts on each other:
Failed to resolve the transaction:
Problem: installed package kmod-nvidia-7.2.11-300.fc45.x86_64-3:615.78.08-1.fc45.x86_64 requires nvidia-kmod-common >= 3:615.78.08, but none of the providers can be installed
- installed package xorg-x11-drv-nvidia-3:615.78.08-1.fc45.x86_64 conflicts with xorg-x11-drv-nvidia provided by nvidia-driver-3:610.43.02-1.fc44.x86_64 from cuda-fedora44-x86_64
- package nvidia-driver-3:610.43.02-1.fc44.x86_64 from cuda-fedora44-x86_64 conflicts with xorg-x11-drv-nvidia provided by xorg-x11-drv-nvidia-3:615.78.08-1.fc45.x86_64 from rpmfusion-nonfree
- conflicting requests
...
- installed package xorg-x11-drv-nvidia-3:615.78.08-1.fc45.x86_64 conflicts with xorg-x11-drv-nvidia provided by nvidia-driver-3:615.71.10-1.fc44.x86_64 from cuda-fedora44-x86_64
...
You can try to add to command line:
--allowerasing to allow removing of installed packages to resolve problems
--skip-broken to skip uninstallable packagesRefusing is the safe outcome. The dangerous one is the hint at the bottom: --allowerasing would rip out the RPM Fusion driver and its akmod, so never add it to a CUDA install on this box. The quieter risk is upgrades. The CUDA repo ships nvidia-settings, nvidia-persistenced and nvidia-modprobe under exactly the names RPM Fusion uses, with the same epoch, so the day NVIDIA’s build is newer than RPM Fusion’s, a plain dnf upgrade will want to pull them from the wrong repo. RPM Fusion’s CUDA howto handles that by hiding the driver packages from the CUDA repo, and I’d treat it as part of the install rather than an extra. The quotes keep your shell from expanding the globs:
sudo dnf config-manager setopt "cuda-fedora44-$(uname -m).exclude=nvidia-driver*,cuda-drivers*,nvidia-modprobe,nvidia-persistenced,nvidia-settings,nvidia-libXNVCtrl*,nvidia-xconfig,nvidia-open,nvidia-imex,nvidia-kmod-common,libnvidia-fbc,kmod-nvidia-*-dkms,nvidia-fs*,nvidia-gds*,xorg-x11-nvidia*"dnf5 doesn’t edit the repo file for this. It writes the exclude to /etc/dnf/repos.override.d/99-config_manager.repo, so editing the repo file won’t lose it. A new repo ID, like the fedora45 one, needs the same command with its own name. I checked the effect on a second Fedora 45 box by asking the CUDA repo for its driver packages before and after:
dnf -q repoquery --repo=cuda-fedora44-x86_64 --latest-limit=1 nvidia-driver nvidia-settings nvidia-persistenced nvidia-modprobe cuda-drivers nvidia-open kmod-nvidia-open-dkmsBefore the exclude, the repo offered seven driver packages:
cuda-drivers-3:610.57.04-1.fc44.x86_64
kmod-nvidia-open-dkms-3:615.71.10-1.fc44.noarch
nvidia-driver-3:615.71.10-1.fc44.x86_64
nvidia-modprobe-3:615.71.10-1.fc44.x86_64
nvidia-open-3:615.71.10-1.fc44.noarch
nvidia-persistenced-3:615.71.10-1.fc44.x86_64
nvidia-settings-3:615.71.10-1.fc44.x86_64After it, the same query printed nothing, while cuda-toolkit was still visible. A dry run of sudo dnf install --assumeno akmod-nvidia xorg-x11-drv-nvidia-cuda cuda-toolkit on that box then resolved every NVIDIA driver package (nvidia-settings, nvidia-persistenced, nvidia-modprobe included) from rpmfusion-nonfree at 615.78.08, which is the result you want.
Installing cuda-toolkit instead of cuda
Install cuda-toolkit, not cuda. The difference on this box was three packages (cuda, cuda-13-4 and cuda-runtime-13-4), and cuda-runtime-13-4 is the piece that depends on a driver. RPM Fusion’s packages satisfied it here, but there’s no reason to let a driver dependency into the transaction at all. The toolkit is big:
sudo dnf install -y cuda-toolkitOn a small root disk, check the dry-run summary first with --assumeno in place of -y. Mine read:
Transaction Summary:
Installing: 75 packages
Total size of inbound packages is 4 GiB. Need to download 4 GiB.
After this operation, 8 GiB extra will be used (install 8 GiB, remove 0 B).In eu-west-1 the download and install took 8 minutes 41 seconds, almost all of it download, and dnf imported NVIDIA’s “Kitmaker (Fedora 44)” signing key on the way. The result lives in /usr/local/cuda-13.4 (5.2 GB) with /usr/local/cuda as an alternatives symlink, so a future 13.5 can sit next to it. The package also drops 000_cuda.conf into /etc/ld.so.conf.d/, and ldconfig -p already resolved libcudart.so.13 without any LD_LIBRARY_PATH. Only PATH needs help:
echo 'export PATH=/usr/local/cuda/bin:$PATH' | sudo tee /etc/profile.d/cuda.sh
source /etc/profile.d/cuda.sh
nvcc --versionnvcc reported the 13.4 release, matching the driver’s UMD ceiling:
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2026 NVIDIA Corporation
Built on Tue_Sep_01_08:45:21_PDT_2026
Cuda compilation tools, release 13.4, V13.4.92
Build cuda_13.4.r13.4/compiler.38855100_0New login shells pick up the profile.d file on their own, so that source is only for the session you are in. Do you need LD_LIBRARY_PATH too? Not with this package, because the ld.so.conf.d entries already cover the libraries.
Compiling a CUDA kernel with nvcc and GCC 16
I expected the compiler to be the problem. Fedora 45 ships GCC 16, and every CUDA release refuses host compilers newer than the ones it was tested with. It turned out CUDA 13.4 had already moved its limit to 16, which you can confirm in the header nvcc checks:
grep -n "__GNUC__ >" /usr/local/cuda/targets/x86_64-linux/include/crt/host_config.hLine 135 only errors out above GCC 16:
135:#if __GNUC__ > 16
139:#endif /* __GNUC__ > 16 */A real test means running code on the GPU, so I wrote a vector add that copies two arrays of a million floats to the card, adds them in a kernel and checks every result on the host. Create the file:
vi ~/vecadd.cuPaste the program exactly as I ran it:
#include <cstdio>
#include <cmath>
#include <cuda_runtime.h>
#define CHECK(call) do { cudaError_t e = (call); if (e != cudaSuccess) { \
fprintf(stderr, "CUDA error: %s at %s:%d\n", cudaGetErrorString(e), __FILE__, __LINE__); return 1; } } while (0)
__global__ void vecAdd(const float *a, const float *b, float *c, int n) {
int i = blockIdx.x * blockDim.x + threadIdx.x;
if (i < n) c[i] = a[i] + b[i];
}
int main() {
const int n = 1 << 20;
size_t bytes = n * sizeof(float);
cudaDeviceProp prop;
CHECK(cudaGetDeviceProperties(&prop, 0));
printf("Device: %s (compute capability %d.%d)\n", prop.name, prop.major, prop.minor);
float *a = (float *)malloc(bytes), *b = (float *)malloc(bytes), *c = (float *)malloc(bytes);
for (int i = 0; i < n; i++) { a[i] = 1.5f * i; b[i] = 2.0f * i; }
float *da, *db, *dc;
CHECK(cudaMalloc(&da, bytes));
CHECK(cudaMalloc(&db, bytes));
CHECK(cudaMalloc(&dc, bytes));
CHECK(cudaMemcpy(da, a, bytes, cudaMemcpyHostToDevice));
CHECK(cudaMemcpy(db, b, bytes, cudaMemcpyHostToDevice));
int threads = 256, blocks = (n + threads - 1) / threads;
vecAdd<<<blocks, threads>>>(da, db, dc, n);
CHECK(cudaGetLastError());
CHECK(cudaMemcpy(c, dc, bytes, cudaMemcpyDeviceToHost));
float maxErr = 0.0f;
for (int i = 0; i < n; i++) maxErr = fmaxf(maxErr, fabsf(c[i] - (a[i] + b[i])));
printf("Added %d elements, max error: %g\n", n, maxErr);
printf("%s\n", maxErr == 0.0f ? "PASS" : "FAIL");
cudaFree(da); cudaFree(db); cudaFree(dc);
free(a); free(b); free(c);
return maxErr == 0.0f ? 0 : 1;
}Compile it with the default GCC 16 and run it:
cd ~
nvcc -o vecadd vecadd.cu
./vecaddCompiling took 1.2 seconds with no warnings, and the kernel matched the host result on all 1,048,576 elements:
Device: NVIDIA A10G (compute capability 8.6)
Added 1048576 elements, max error: 0
PASSOne screen holds the whole check, next to the driver and compiler versions it ran against.
ldd ./vecadd lists no libcudart, because nvcc links the CUDA runtime statically by default. That detail matters in the container section: the binary only needs the driver’s libcuda.so at run time, which is exactly what CDI injects.
Older toolkits are a different story. If you pin a CUDA release whose host_config.h stops below 16, Fedora 45 carries a GCC 15 compatibility compiler, and I confirmed nvcc accepts it as the host compiler:
sudo dnf install -y gcc15-c++
nvcc -ccbin g++-15 -o vecadd vecadd.cuWith GCC 15 as host compiler, the build passed the same test. I’d reach for -ccbin before -allow-unsupported-compiler, because the second flag only silences the version check and leaves you guessing when a template error shows up later.
Error: “CMAKE_CUDA_ARCHITECTURES is not set” when building deviceQuery
I also wanted NVIDIA’s own deviceQuery sample, and it failed twice before it ran. The samples now live under cpp/ instead of the old Samples/ path that most guides still quote, and a standalone sample build refuses to guess your GPU architecture:
CMake Error at /home/fedora/cuda-samples/cmake/DetectCudaArch.cmake:2 (message):
CMAKE_CUDA_ARCHITECTURES is not set.
For standalone sample builds, specify a target architecture explicitly:
cmake -DCMAKE_CUDA_ARCHITECTURES=<arch> <path-to-sample>Pass the compute capability without the dot, so 86 for the A10G’s 8.6 from the vecadd output. These are the steps that built it in 4 seconds:
sudo dnf install -y cmake git make
git clone --depth 1 https://github.com/NVIDIA/cuda-samples.git
cd cuda-samples/cpp/1_Utilities/deviceQuery
mkdir build && cd build
cmake -DCMAKE_CUDA_ARCHITECTURES=86 ..
make
./deviceQueryRead the version pair near the top and the verdict at the bottom.
Device 0: "NVIDIA A10G"
CUDA Driver Version / Runtime Version 13.4 / 13.4
CUDA Capability Major/Minor version number: 8.6
Total amount of global memory: 22640 MBytes (23740153856 bytes)
(080) Multiprocessors, (128) CUDA Cores/MP: 10240 CUDA Cores
GPU Max Clock rate: 1710 MHz (1.71 GHz)
...
Memory Bus Width: 384-bit
Device has ECC support: Enabled
...
deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 13.4, CUDA Runtime Version = 13.4, NumDevs = 1
Result = PASSMatching driver and runtime numbers (13.4 and 13.4 here) mean a healthy install. A runtime number higher than the driver number means the toolkit outran the driver, and the fix is a driver update, not a toolkit downgrade.
Installing the NVIDIA Container Toolkit for CDI
Containers don’t need the CUDA Toolkit at all, so you can do this part on a box that only has the driver. NVIDIA ships one repo file that works for every RPM distro:
curl -s -L https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo | sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo
sudo dnf install -y nvidia-container-toolkit
nvidia-ctk --versionFour packages, 11 MiB, done in 4 seconds:
NVIDIA Container Toolkit CLI version 1.20.1
commit: dffc40b4f820ce5c512633bac9e0418d0e05a2eeThe install put more on the box than the CLI. These are the pieces that matter for CDI:
nvidia-ctk, from nvidia-container-toolkit-basenvidia-cdi-refresh.pathand.service, enabled by preset- the udev rule
99-nvidia-cdi-refresh.rules /etc/nvidia-container-toolkit/nvidia-cdi-refresh.envfor overrides
Most guides tell you to run nvidia-ctk cdi generate next. I did, then realised it was redundant. Since toolkit 1.18 the package enables nvidia-cdi-refresh.path, which had already written /var/run/cdi/nvidia.yaml while the install transaction was still running. It regenerates the file on toolkit and driver installs or upgrades and on reboot, but not when a driver is removed, so rerun it by hand after an uninstall. List what it found:
nvidia-ctk cdi listThree names point at the one GPU: by index, by UUID and the all alias:
nvidia.com/gpu=0
nvidia.com/gpu=GPU-0b18b197-4ea9-4e4d-71bd-739e2e3bf67a
nvidia.com/gpu=all
time="2026-10-27T05:51:00Z" level=info msg="Found 3 CDI devices"Expect the service log to be full of Could not locate warnings about X11 files such as nvidia_drv.so and libglxserver_nvidia.so. On a headless box with no X server those files don’t exist, and the spec is still complete for compute. A hand-generated copy in /etc/cdi is the thing I’d avoid: it didn’t create duplicate devices in my test, but after the next driver update only the /var/run/cdi copy gets refreshed. I haven’t tested which copy wins when they disagree, and I’d rather not find out on a box that matters. If you ever need a manual refresh, restart the service instead:
sudo systemctl restart nvidia-cdi-refresh.serviceNVIDIA’s CDI support page documents the refresh units and the environment file at /etc/nvidia-container-toolkit/nvidia-cdi-refresh.env if you need to change where the spec lands.
Running GPU containers in rootless Podman
Podman 6.1 comes with the Fedora Cloud image and reads CDI specs natively, so the only flag a container needs is --device nvidia.com/gpu=all. My first rootless attempt still failed:
podman run --rm --device nvidia.com/gpu=all docker.io/nvidia/cuda:13.4.3-base-ubi9 nvidia-smiWhy would NVML fail when the device nodes were clearly injected? The image pulled fine and the error came from inside the container.
Failed to initialize NVML: Insufficient PermissionsAdding --security-opt=label=disable to that exact command printed the full table, which pointed at SELinux rather than at the GPU or the spec.
Fix “Failed to initialize NVML: Insufficient Permissions” with SELinux enforcing
SELinux is the cause, not file permissions. Fedora’s container policy stops a confined container process from using host device nodes unless the container_use_devices boolean is on, and it was off by default on this image. NVIDIA’s docs answer this with --security-opt=label=disable, which works but drops SELinux separation for that container entirely. The boolean is the narrower fix: containers keep their labels and only gain device access. Check it, then turn it on persistently:
getsebool container_use_devices
sudo setsebool -P container_use_devices onAfter that, the same rootless command printed the full nvidia-smi table from inside the UBI 9 image, and so did the vecadd binary from the host, mounted into a container that has no CUDA Toolkit in it:
podman run --rm --device nvidia.com/gpu=all -v ~/vecadd:/vecadd:Z docker.io/nvidia/cuda:13.4.3-base-ubi9 /vecaddBoth runs, with the failure before the boolean, look like this:
Rootful sudo podman run with the same flag worked too once the boolean was on. If your containers run under Quadlet units, the Podman on Fedora guide covers the unit side; the GPU part is one AddDevice=nvidia.com/gpu=all line in the [Container] section, which I have not tested on this box.
Other errors Podman printed and what they meant
I broke the command on purpose a few ways to see what each mistake looks like, because the messages don’t all point at the GPU:
- No
--deviceflag:crun: executable file `nvidia-smi` not found in $PATH - vecadd with no
--device:CUDA driver version is insufficient for CUDA runtime version - A GPU index that doesn’t exist:
Error: setting up CDI devices: unresolvable CDI devices nvidia.com/gpu=1
Watch for the second one. It reads like a version mismatch, but the container simply had no libcuda.so because nothing injected the driver, so check the flag before you go hunting for a newer driver. The image doesn’t ship nvidia-smi either, which is why the first error says “not found” instead of anything about GPUs. To pin a container to one card on a multi-GPU box, use nvidia.com/gpu=0 or the UUID form from nvidia-ctk cdi list.
Running GPU containers in Docker with CDI
Docker surprised me in the other direction. I’d planned to run nvidia-ctk runtime configure and edit daemon.json, and it turned out Docker 29.9 needed neither. Docker CE published a fedora 45 tree by the time I ran this, so the standard repo file worked as is (the Docker on Fedora guide has the full install and the podman-docker conflict):
sudo dnf config-manager addrepo --from-repofile=https://download.docker.com/linux/fedora/docker-ce.repo
sudo dnf install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin
sudo systemctl enable --now docker
sudo docker infoWith no config file at all, docker info already listed the CDI directories and the GPU the refresh service had described:
Server Version: 29.9.2
CDI spec directories:
/etc/cdi
/run/cdi
Discovered Devices:
cdi: nvidia.com/gpu=0
cdi: nvidia.com/gpu=GPU-0b18b197-4ea9-4e4d-71bd-739e2e3bf67a
cdi: nvidia.com/gpu=allBoth the CDI flag and the familiar --gpus all worked on that untouched daemon:
sudo docker run --rm --device nvidia.com/gpu=all docker.io/nvidia/cuda:13.4.3-base-ubi9 nvidia-smi -L
sudo docker run --rm --gpus all docker.io/nvidia/cuda:13.4.3-base-ubi9 nvidia-smi -LEach run printed the same line with the card’s UUID:
GPU 0: NVIDIA A10G (UUID: GPU-0b18b197-4ea9-4e4d-71bd-739e2e3bf67a)In the terminal from that part of the session, Docker found the CDI devices on its own.
I did run sudo nvidia-ctk runtime configure --runtime=docker afterwards to compare. It added an nvidia runtime to /etc/docker/daemon.json, and every test passed the same way before and after, including the vecadd binary. You only need that runtime for images or tools that still ask for --runtime=nvidia with NVIDIA_VISIBLE_DEVICES. One caveat on SELinux: container_use_devices was already on from the Podman section when I ran these, so I cannot tell you whether Docker would have hit the same NVML error without it.
Putting Ollama on the GPU through Podman
Most people want this stack for a local model server, so I finished with Ollama in rootless Podman. It’s the same --device flag and nothing else:
podman run -d --name ollama --device nvidia.com/gpu=all -p 11434:11434 docker.io/ollama/ollama
podman exec ollama ollama run llama3.2:1b "say hi"
podman exec ollama ollama psRead the PROCESSOR column. “100% GPU” means every layer was offloaded; a CPU percentage there means the CDI device didn’t reach the container:
NAME ID SIZE PROCESSOR CONTEXT RUNNER UNTIL
llama3.2:1b baf6a787fdff 1.5 GB 100% GPU 4096 llamacpp 4 minutes from now Ollama logged library=CUDA compute=8.6 name=CUDA0 description="NVIDIA A10G" and loaded its cuda_v13 backend, and host nvidia-smi showed llama-server holding 1738 MiB. If you’ve been running models on CPU from the Ollama on Fedora CPU guide, this container is the drop-in GPU version, and the GPU buying guide for LLMs covers how much VRAM bigger models need.
Removing the toolkit later is one command. A dry run of sudo dnf remove "cuda-*13-4*" listed 74 packages and 8 GiB freed. I only read its summary, not the full package list, so run it with --assumeno first and check that nothing from rpmfusion-nonfree is in it. The small cuda-toolkit-config-common packages don’t match that glob and stay behind, and gcc-c++ came in as a toolkit dependency, so expect dnf to offer to remove the C++ compiler with it.
What I still don’t know
A cloud box answered the questions I started with, and left a few I couldn’t settle there.
- Secure Boot: off on AWS, so the MOK signing path is untested here
- Fedora 44 on a GPU: same repo, but I only ran the GPU half on 45
- The fedora45 CUDA repo: not published yet, so no swap tested
- Desktop GPUs with a display attached: the CDI warnings would differ
Driver drift is the one I’d watch. On test day RPM Fusion was a point release ahead of NVIDIA’s own 615 build, so the exclude had nothing to protect against yet, and I couldn’t see what an unprotected dnf upgrade does when NVIDIA’s repo is the one that’s ahead. With the exclude in place it shouldn’t matter, because the CUDA repo no longer offers any package that collides with RPM Fusion’s. I’d still run sudo dnf upgrade --assumeno after each NVIDIA driver release and read where the nvidia packages come from before letting it apply.
