Ollama said the model sat 99% on the GPU, yet the card was only 31% busy and drawing 94 W out of a 300 W budget. The nvidia-smi command is how you catch that kind of gap, because it reads the driver directly instead of trusting what an app thinks it loaded. It’s also the quickest way to monitor GPU usage on Linux, check VRAM per process, and see whether the clocks are being held back.
You’ll get the default table decoded field by field, live monitoring, CSV logs you can load into a spreadsheet, dmon and pmon, and tested fixes for the errors people search for. Every command and output below was run in October 2026 on a rented RTX 5070 Ti with NVIDIA driver 595.84 (CUDA 13.2) on Ubuntu 24.04, with Ollama serving Qwen3 models so the numbers come from real load, not an idle card.
Read the default nvidia-smi output
Run it with no arguments and you get one snapshot, with a header for the driver, a row per GPU and a process table at the bottom. It works as a normal user, so you don’t need sudo just to read stats, and that matters when you hand GPU access to people who shouldn’t have root on the box.
nvidia-smiThis is the card partway into a long Qwen3 8B generation. GPU-Util is pinned near 100%, the board is pulling 205 W, and one llama-server process owns almost all of the 5.6 GB in use:
Mon Oct 26 04:35:47 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 595.84 Driver Version: 595.84 CUDA Version: 13.2 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 5070 Ti On | 00000000:09:00.0 Off | N/A |
| 30% 47C P1 205W / 300W | 5638MiB / 16303MiB | 97% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 1046 C ...local/lib/ollama/llama-server 5628MiB |
+-----------------------------------------------------------------------------------------+The header’s “CUDA Version” trips people up. It’s the newest CUDA runtime the driver supports, not the toolkit you installed, so a box can show 13.2 here and still have no nvcc at all. The rest of the table decodes like this:
| Field | Value above | What it means |
|---|---|---|
| Persistence-M | On | Driver stays loaded with no clients, so the first query doesn’t pay a load delay |
| Bus-Id | 00000000:09:00.0 | PCI address; stable across reboots, unlike the index |
| Disp.A | Off | Display Active: whether a display is initialized on the GPU |
| Volatile Uncorr. ECC | N/A | Uncorrected ECC errors since driver load; N/A on GeForce cards without ECC |
| Fan | 30% | Intended fan speed, not a measured RPM |
| Temp | 47C | Core GPU temperature |
| Perf | P1 | Performance state, P0 (maximum) to P12 (minimum) |
| Pwr:Usage/Cap | 205W / 300W | Board power draw against the enforced power limit |
| Memory-Usage | 5638MiB / 16303MiB | VRAM used across all processes, plus driver overhead |
| GPU-Util | 97% | Share of the sample period when at least one kernel was running |
| Compute M. | Default | Compute mode; Default allows several processes per GPU |
| MIG M. | N/A | Multi-Instance GPU mode; N/A on cards without MIG support, GeForce included |
GPU-Util is the column people misread most. It says how often a kernel was running, not how hard the SMs were working, so a single tiny kernel looping forever can show 100%. Treat it as “busy or idle” and pair it with power draw, which tracks real work much better. A card at 97% and 205 W is genuinely loaded; a card at 97% and 40 W isn’t.
For a quick inventory, nvidia-smi -L lists each GPU with its UUID. Use that UUID in scripts rather than the index, because NVIDIA doesn’t guarantee the index order stays the same across reboots, and on a multi-GPU box a reshuffle sends your job to the wrong card.
nvidia-smi -LWith one GPU in this box you get one line back. On a multi-GPU server you get one line per card, so this is the list to check first when a job lands on the wrong GPU. Copy the UUID from here, because that’s the value you’ll pass to -i later.
GPU 0: NVIDIA GeForce RTX 5070 Ti (UUID: GPU-a7a3031a-b037-e253-1c99-793c29f4bd9d)Record the driver and NVML versions before you compare numbers between machines. Field names and defaults move between driver branches, so a query that works on one host can fail on another, and the version line is the first thing to check when it does.
nvidia-smi --versionAll four lines should agree on the driver branch. If NVML and the driver differ, you’ll hit the version mismatch error covered near the end.
NVIDIA-SMI version : 595.84
NVML version : 595.84
DRIVER version : 595.84
CUDA Version : 13.2Paste that block into any bug report or forum post about GPU behaviour, because the first reply you’ll get otherwise is a request for the driver version.
Run nvidia-smi in a live loop
You have two ways to refresh the table, and they behave differently. watch redraws the screen in place, which is what you want on a terminal you’re staring at while a job runs.
watch -n 1 nvidia-smiThe watch binary comes from procps, which ships on Ubuntu and most other distros. The built-in loop flag prints a fresh table every N seconds and scrolls, which suits piping to a file or running inside tmux logging. With no number, -l defaults to 5 seconds:
nvidia-smi -l 1Press Ctrl+C to stop it. For anything you will read later, skip the full table and loop over a short query instead. A full table every second is about 20 lines of mostly unchanged text, while a CSV query is one line you can sort, graph and diff, so an hour of logging stays small enough to open in a spreadsheet.
Get GPU fields as CSV with –query-gpu
--query-gpu takes a comma-separated list of fields, and --format=csv is mandatory with it. This one covers what most people want when they monitor GPU usage on Linux: utilization, memory, temperature, power and clocks.
nvidia-smi --query-gpu=index,name,utilization.gpu,utilization.memory,memory.used,memory.total,temperature.gpu,power.draw,power.limit,clocks.sm,clocks.mem,pstate --format=csvUnder the same 8B load, the SM clock holds at 2872 MHz and device memory is busy 81% of the time, because token generation reads the model weights out of VRAM for every token it produces. That memory figure is the one to watch on LLM boxes, since generation speed tracks memory bandwidth far more than SM load.
index, name, utilization.gpu [%], utilization.memory [%], memory.used [MiB], memory.total [MiB], temperature.gpu, power.draw [W], power.limit [W], clocks.current.sm [MHz], clocks.current.memory [MHz], pstate
0, NVIDIA GeForce RTX 5070 Ti, 97 %, 81 %, 5638 MiB, 16303 MiB, 48, 240.90 W, 300.00 W, 2872 MHz, 13801 MHz, P1Notice the header expanded clocks.sm to clocks.current.sm; short names are aliases. The driver used for this guide accepts 206 fields, and nvidia-smi --help-query-gpu prints every one with a description. Add noheader,nounits to the format when a script will parse the result, since it drops the header row and the “MiB” and “W” suffixes.
Power has three fields, and it’s worth knowing which one you’re logging. On this driver power.draw returned the same value as power.draw.average, while power.draw.instant is the momentary reading:
nvidia-smi --query-gpu=power.draw,power.draw.average,power.draw.instant,power.limit,enforced.power.limit,power.max_limit --format=csvThe card’s limit is 300 W with a ceiling of 350 W, and under this load it drew about 242 W. So this workload leaves roughly 60 W of headroom, which matters later when you check whether the power limit is what’s slowing a job down. A card pinned at its limit tells a different story, because then the driver trims clocks to stay under it.
power.draw [W], power.draw.average [W], power.draw.instant [W], power.limit [W], enforced.power.limit [W], power.max_limit [W]
242.03 W, 242.03 W, 242.63 W, 300.00 W, 300.00 W, 350.00 WDon’t read much into one sample. The default table caught 205 W and the first dmon row 206 W, while most readings after that sat around 240 W. Log the average over time for trends, and reach for the instant value only when you’re hunting spikes.
Log GPU metrics to a CSV file
Combine a query with -l and -f and nvidia-smi becomes a lightweight logger that drops straight into a spreadsheet. The run below captured 60 seconds of the Qwen3 8B load:
- Start the logger with a one-second interval and a file name.
- Run the workload you want to measure.
- Stop the logger, then summarize the file.
The timeout 60 prefix ends the loop on its own; drop it and press Ctrl+C when you are done. Be careful with the file name, because -f overwrites the file each time it starts and you will lose the previous run.
timeout 60 nvidia-smi --query-gpu=timestamp,utilization.gpu,memory.used,temperature.gpu,power.draw,clocks.sm,clocks_event_reasons.active --format=csv,noheader,nounits -l 1 -f gpu-log.csvCheck the first lines before you trust a long run. A typo in a field name fails the whole query, so it’s cheaper to find out after five seconds than after an overnight test.
head -5 gpu-log.csvEach row is a timestamp followed by bare numbers, so it imports into a spreadsheet or pandas without any cleanup. The second row’s 206 W dip isn’t a one-off: the full log dips to 205 to 208 W at rows 2, 17, 32 and 46, roughly every 15 seconds, which is most likely the load script starting a new request. That’s why you judge a log by its averages and only act on a pattern that holds for several samples.
2026/10/26 04:37:00.021, 97, 5638, 55, 246.28, 2857, 0x0000000000000000
2026/10/26 04:37:01.027, 96, 5638, 54, 206.22, 2872, 0x0000000000000000
2026/10/26 04:37:02.027, 96, 5638, 55, 243.77, 2857, 0x0000000000000000
2026/10/26 04:37:03.027, 96, 5638, 55, 243.99, 2857, 0x0000000000000000
2026/10/26 04:37:04.027, 96, 5638, 56, 244.48, 2857, 0x0000000000000000To boil a run down to one line, save a small awk script. It’s handy when you compare runs, for example the same prompt at two power limits, because you get one summary line per log instead of scrolling through thousands of rows.
vim summarize.awkPaste this in. The field numbers follow the column order of the query above, so change them if you reorder fields.
BEGIN { FS = ", " }
{ n++; util += $2; pw += $5; if ($5 > maxpw) maxpw = $5; if ($4 > maxt) maxt = $4; if ($3 > maxmem) maxmem = $3 }
END { printf "samples=%d avg_util=%.1f%% avg_power=%.1fW max_power=%.1fW max_temp=%dC peak_mem=%dMiB\n", n, util/n, pw/n, maxpw, maxt, maxmem }Then point it at the log file you just wrote. The script reads nothing but the CSV, so you can run it on a laptop after copying the file off the GPU box. That keeps a rented GPU’s billable time for the workload itself.
awk -f summarize.awk gpu-log.csvSixty samples, 96% average utilization, and the temperature crept from 55 to 58 °C over the minute. The clock event column stayed at all zeros the whole time, so nothing held the clocks back:
samples=60 avg_util=96.1% avg_power=243.6W max_power=248.3W max_temp=58C peak_mem=5638MiBWhen one second is too coarse, swap -l for -lms and give it milliseconds. NVIDIA warns that faster polling can cost performance, so keep sub-second runs short:
timeout 2 nvidia-smi --query-gpu=timestamp,utilization.gpu,power.draw --format=csv,noheader -lms 200You get five rows a second, 200 ms apart. Ten rows is plenty to see the pattern, so there’s no reason to leave a 200 ms loop running for long. Go back to -l 1 for anything longer than a minute, because a file at five rows a second grows fast and adds little.
2026/10/26 04:38:10.110, 96 %, 249.01 W
2026/10/26 04:38:10.317, 96 %, 248.90 W
2026/10/26 04:38:10.517, 96 %, 248.90 W
2026/10/26 04:38:10.717, 96 %, 248.90 W
2026/10/26 04:38:10.917, 97 %, 248.23 W
2026/10/26 04:38:11.117, 97 %, 248.23 W
2026/10/26 04:38:11.317, 97 %, 247.90 W
2026/10/26 04:38:11.517, 97 %, 247.90 W
2026/10/26 04:38:11.717, 97 %, 247.90 W
2026/10/26 04:38:11.917, 97 %, 247.73 WThe power value repeats across two or three samples, so in this run the reading only changed every 400 to 600 ms. Polling faster than the value refreshes just gives you duplicate rows. For a logger that has to survive driver upgrades, NVIDIA says nvidia-smi’s output isn’t guaranteed to stay backward compatible and points to NVML and its nvidia-ml-py Python bindings instead. For a fleet with dashboards and alerts, the DCGM, Prometheus and Grafana stack takes over where a CSV file stops.
List per-process GPU memory
The process table at the bottom of the default output truncates long paths. --query-compute-apps gives the same data in full, one row per process, which is what you’d grep in a script:
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csvThe Ollama build in this lab runs the model inside a llama-server child process, so that’s the name you’ll see, not ollama:
pid, process_name, used_gpu_memory [MiB]
1046, /usr/local/lib/ollama/llama-server, 5628 MiBThe 10 MiB gap between this 5628 MiB and the 5638 MiB total in Memory-Usage isn’t attributed to any process, so don’t expect the per-process rows to add up exactly. For per-process activity over time, pmon adds SM and memory utilization per PID, and -s um adds the framebuffer column:
nvidia-smi pmon -s um -c 3You see three one-second samples of the same process, with its 5628 MB of framebuffer in the fb column. On a shared box with several jobs, this is the quickest way to tell which PID is eating the SMs, because the default table only shows memory per process.
# gpu pid type sm mem enc dec jpg ofa fb ccpm command
# Idx # C/G % % % % % % MB MB name
0 1046 C 96 80 - - - - 5628 0 llama-server
0 1046 C 96 80 - - - - 5628 0 llama-server
0 1046 C 96 80 - - - - 5628 0 llama-server Containers are where per-process rows go missing. This lab ran inside a Docker container with its own PID namespace (PID 1 was sh and ps -e printed 13 lines), and on this driver the row still appeared with the container’s own PID, matching ps. Other setups aren’t so lucky: users on the NVIDIA developer forum report container rows that are missing or show host PIDs with “[Not Found]” names. If that’s what you see, run nvidia-smi on the host (sharing the host PID namespace with --pid=host is the usual workaround, not tested here), and trust memory.used over the process list.
Use nvidia-smi dmon to monitor over time
dmon prints one compact line per second per GPU, which is easier to eyeball than repeated tables. With no options it shows power, temperature, utilization and clocks (the puc groups). -c stops after a set number of samples:
nvidia-smi dmon -c 5The sm column holds at 96 to 97% and pwr climbs from 206 to 242 W as the card settles into the job.
# gpu pwr gtemp mtemp sm mem enc dec jpg ofa mclk pclk
# Idx W C C % % % % % % MHz MHz
0 206 49 - 96 81 0 0 0 0 13801 2872
0 240 50 - 96 81 0 0 0 0 13801 2865
0 240 50 - 97 81 0 0 0 0 13801 2872
0 241 50 - 96 80 0 0 0 0 13801 2872
0 242 50 - 97 81 0 0 0 0 13801 2872 You pick metric groups with -s, and the letters combine freely. These are the ones the driver’s own help lists, and on a single-GPU workstation pucm is the set you will use most because it adds the memory you care about to the default view:
ppower and temperatureuutilizationcSM and memory clocksvpower and thermal violationsmframebuffer and BAR1 memoryeECC and PCIe replay errorstPCIe throughput
Add -o T for a time column, which you will want the moment you compare the output against an application log.
nvidia-smi dmon -s pucm -o T -c 5Now each row carries a timestamp, and the fb column shows the 5638 MB in use. Add -f with a file name and the same lines go to disk, which turns dmon into a logger for people who prefer columns over CSV.
#Time gpu pwr gtemp mtemp sm mem enc dec jpg ofa mclk pclk fb bar1 ccpm
#HH:MM:SS Idx W C C % % % % % % MHz MHz MB MB MB
04:36:07 0 242 50 - 96 81 0 0 0 0 13801 2872 5638 4 0
04:36:08 0 241 50 - 96 81 0 0 0 0 13801 2872 5638 4 0
04:36:09 0 243 51 - 96 81 0 0 0 0 13801 2872 5638 4 0
04:36:10 0 243 51 - 95 80 0 0 0 0 13801 2872 5638 4 0
04:36:11 0 244 51 - 97 81 0 0 0 0 13801 2872 5638 4 0 The mtemp dash means this GeForce card doesn’t expose a memory temperature sensor. dmon also takes -f to write to a file and -d to change the interval, and its built-in help says it covers up to 16 GPUs, while pmon -h on this driver says 4.
Check power, temperature and throttling
When a job runs slower than expected, the first question is whether the clocks were pulled down, and the driver records why. -q -d prints the detailed sections for just the topics you name:
nvidia-smi -q -d POWER,TEMPERATURE,PERFORMANCEThis was taken under the 8B load. Every clock event reason reads Not Active, and the power section shows NVIDIA averaging 119 samples over 2.36 seconds:
==============NVSMI LOG==============
Timestamp : Mon Oct 26 04:36:30 2026
Driver Version : 595.84
CUDA Version : 13.2
Attached GPUs : 1
GPU 00000000:09:00.0
Performance State : P1
Clocks Event Reasons
Idle : Not Active
Applications Clocks Setting : Not Active
SW Power Cap : Not Active
HW Slowdown : Not Active
HW Thermal Slowdown : Not Active
HW Power Brake Slowdown : Not Active
Sync Boost : Not Active
SW Thermal Slowdown : Not Active
Display Clock Setting : Not Active
Clocks Event Reasons Counters
SW Power Capping : 58038447 us
Sync Boost : 0 us
SW Thermal Slowdown : 0 us
HW Thermal Slowdown : 0 us
HW Power Braking : 0 us
Sparse Operation Mode : N/A
Temperature
GPU Current Temp : 53 C
GPU T.Limit Temp : 35 C
GPU Shutdown T.Limit Temp : -5 C
GPU Slowdown T.Limit Temp : -2 C
GPU Max Operating T.Limit Temp : 0 C
GPU Target Temperature : N/A
Memory Current Temp : N/A
Memory Max Operating T.Limit Temp : N/A
GPU Power Readings
Average Power Draw : 245.34 W
Instantaneous Power Draw : 245.38 W
Current Power Limit : 300.00 W
Requested Power Limit : 300.00 W
Default Power Limit : 300.00 W
Min Power Limit : 250.00 W
Max Power Limit : 350.00 W
Power Samples
Duration : 2.36 sec
Number of Samples : 119
Max : 248.62 W
Min : 232.62 W
Avg : 245.28 W
GPU Memory Power Readings
Average Power Draw : N/A
Instantaneous Power Draw : N/A
Module Power Readings
Average Power Draw : N/A
Instantaneous Power Draw : N/A
Current Power Limit : N/A
Requested Power Limit : N/A
Default Power Limit : N/A
Min Power Limit : N/A
Max Power Limit : N/A
EDPp Multiplier : N/A
Most of that block reads as you would expect, but three lines in it mislead people who skim, and each one has sent someone chasing a problem that does not exist:
- T.Limit is a margin, not a temperature. 35 C means 35 degrees of headroom before the maximum operating temperature, and the -2 C slowdown and -5 C shutdown thresholds count down from that same margin.
- The SW Power Capping counter is history. It’s a running total in microseconds, so 58 seconds says nothing about the job running now.
- Idle is deprecated. It read Not Active even when the card sat at 195 MHz doing nothing later on, so don’t build an idle check on it.
The counter is the useful one once you read it as a delta, which the next check does.
To check whether throttling is happening right now, read the counter twice and compare. These reads were taken 30 seconds apart under the 8B load:
nvidia-smi --query-gpu=clocks_event_reasons_counters.sw_power_cap,clocks_event_reasons_counters.sw_thermal_slowdown,clocks_event_reasons_counters.hw_thermal_slowdown --format=csvThe first read gives the baseline for all three counters:
clocks_event_reasons_counters.sw_power_cap [us], clocks_event_reasons_counters.sw_thermal_slowdown [us], clocks_event_reasons_counters.hw_thermal_slowdown [us]
58038447 us, 0 us, 0 usThirty seconds later, read the power cap counter again, this time alongside utilization and power so you know the card was really working in between:
nvidia-smi --query-gpu=clocks_event_reasons_counters.sw_power_cap,utilization.gpu,power.draw --format=csv,noheaderThe counter didn’t move while the card ran at 97% and 245 W, so it wasn’t capped at any point in that window:
58038447 us, 97 %, 245.74 WFor a one-shot check, the clocks_event_reasons.* fields return a bitmask plus a readable flag per reason. A mask of all zeros means the clocks are running as high as they’re allowed to:
nvidia-smi --query-gpu=clocks_event_reasons.active,clocks_event_reasons.sw_power_cap,clocks_event_reasons.hw_slowdown,clocks_event_reasons.hw_thermal_slowdown,clocks_event_reasons.sw_thermal_slowdown --format=csvNothing was flagged on this card under load. That fits the counter delta above, so both views agree that the clocks weren’t being held back. When they disagree, trust the counter delta, because the flags only show the instant you asked.
clocks_event_reasons.active, clocks_event_reasons.sw_power_cap, clocks_event_reasons.hw_slowdown, clocks_event_reasons.hw_thermal_slowdown, clocks_event_reasons.sw_thermal_slowdown
0x0000000000000000, Not Active, Not Active, Not Active, Not ActiveA 400-sample run at 100 ms intervals, covering the 40 seconds after the 8B model started loading, never caught a single non-zero mask either. That’s the real finding for this card: Qwen3 8B on it peaked at 249 W and never reached the 300 W limit. Older guides call these “clocks throttle reasons”; NVIDIA renamed them to clocks event reasons and kept the old clocks_throttle_reasons.* query names only for backward compatibility, so use the new names in anything you write now. If SW Power Cap does show Active on your card, the power limit is the bottleneck. Changing it needs root on the host:
nvidia-smi -pl 250Inside a rented container, even as root, the driver refuses it, because the container can’t manage the physical device:
Failed to set power management limit for GPU 00000000:09:00.0: Insufficient Permissions
Terminating early due to previous errors.On your own box the value has to fall between the Min Power Limit and Max Power Limit from the -q -d POWER output, which is 250 to 350 W on this card. A successful change couldn’t be tested here, so confirm on your own box whether it survives a reboot.
Check what Ollama really holds in VRAM
ollama ps reports where a model is loaded, and for a model that doesn’t fit it can look healthier than it is. Qwen3 30B is an 18 GB download, which can’t fit in 16 GB of VRAM, so it was loaded next. Check what Ollama says first:
ollama psIt reports 15 GB loaded and 99% of it on the GPU, which reads like a model that fits.
NAME ID SIZE PROCESSOR CONTEXT RUNNER UNTIL
qwen3:30b ad815644918f 15 GB 1%/99% CPU/GPU 4096 llamacpp 4 minutes from now Now ask the driver what the card is actually doing while that model generates. If Ollama were right, you’d expect numbers close to the 8B run from earlier.
nvidia-smi --query-gpu=utilization.gpu,memory.used,memory.total,power.draw,clocks.sm --format=csvVRAM is nearly full at 14,802 MiB, but the GPU is busy only 31% of the time and draws 94 W:
utilization.gpu [%], memory.used [MiB], memory.total [MiB], power.draw [W], clocks.current.sm [MHz]
31 %, 14802 MiB, 16303 MiB, 93.80 W, 2880 MHzOllama’s server log explains the gap. The runner put all 49 layers on the GPU but marked 11 of them as “overflowing”, which for this mixture-of-experts model means part of each of those layers’ expert weights stays in system RAM and gets computed on the CPU:
common_params_fit_impl: - CUDA0 (NVIDIA GeForce RTX 5070 Ti): 49 layers (11 overflowing), 14492 MiB used, 1115 MiB freePut the two runs side by side on the same card and the difference is plain. Every row comes from the captures above or the Ollama server log of the same session.
| Measurement | qwen3:8b (fits) | qwen3:30b (spills) |
|---|---|---|
Model size on disk (ollama list) | 5.2 GB | 18 GB |
| nvidia-smi memory.used | 5638 MiB | 14802 MiB |
| nvidia-smi GPU-Util | 84 to 97% (96.1% average) | 29 to 32% |
| Power draw | 205 to 249 W (243.6 W average) | 79 to 94 W |
| Generation speed (Ollama log) | about 140 tokens/s | about 83 tokens/s |
The 99% figure is about how many bytes are where, not about where the time goes. A small slice of weights on the CPU makes the GPU wait on every token, and only nvidia-smi’s utilization and power columns show that waiting. So when a model feels slow, check GPU-Util under load: a figure near 100% means the card is the limit, while a figure around 30% with full VRAM means you’re paying for CPU offload. A smaller quantization, a shorter context, or a card with more VRAM fixes it. The Ollama commands cheat sheet covers the context and keep-alive settings, and the GPU guide for local LLMs maps model sizes to cards.
The capture below puts the 8B run from the top of the page next to the full nvidia-smi table during the 30B run, so you can compare the same card under the two models at a glance.
Memory-Usage nearly maxed out with GPU-Util stuck near a third is the pattern to look for. Don’t expect free to warn you either: during the 30B run it showed only 2.3 GB of RAM used and 29 GB sitting in buff/cache, so the CPU-side weights never look like a memory problem.
Fix common nvidia-smi errors
Three errors account for most failed runs. The first was reproduced on the rented box and the second partly, but the third couldn’t be reproduced in a container, so its section sticks to what you should check. The exit code is useful in scripts: NVIDIA documents 6 as “a query to find an object was unsuccessful” and 9 as “NVIDIA driver is not loaded”.
Failed to initialize NVML: Driver/library version mismatch
This means the userspace NVML library and the loaded kernel module come from different driver releases. The usual trigger is a package upgrade that replaced the libraries while the old module stayed in memory. To reproduce it, the lab ran an nvidia-smi and NVML from the 580 driver branch against the running 595 module:
LD_LIBRARY_PATH=/opt/old/root/usr/lib/x86_64-linux-gnu /opt/old/root/usr/bin/nvidia-smiThe 580 nvidia-smi prints the library version on a second line, which tells you which side is stale. The command exited with code 18:
Failed to initialize NVML: Driver/library version mismatch
NVML library version: 580.196Compare it with the version of the module that is actually loaded. This file comes from the kernel module itself, so it still answers when NVML refuses to start.
cat /proc/driver/nvidia/versionKernel side 595.84, library 580.196: the library was the old half. Whichever version doesn’t match what you meant to install is the one to fix, so read both before you touch anything.
NVRM version: NVIDIA UNIX Open Kernel Module for x86_64 595.84 Release Build (dvs-builder@U22-I3-AM25-26-2) Wed Jun 10 21:06:37 UTC 2026
GCC version: gcc version 12.3.0 (Ubuntu 12.3.0-1ubuntu1~22.04.3) In the lab, dropping the stale library path brought nvidia-smi straight back. On a real host the fix is to get both halves onto the same release: after an apt or dnf driver upgrade, reboot so the new module loads. If the library side is the older one, finish the interrupted upgrade instead. Leftover libraries from a .run installer cause the same error, and the guide to removing manually installed NVIDIA drivers cleans those out.
No devices were found
This one means the driver loaded and answered, but no GPU matched the request. The easiest way to see it is to ask for a GPU index that doesn’t exist, which on this single-GPU box returned exit code 6:
nvidia-smi -i 1You get the same message you’d see with no visible GPU at all, so the text alone doesn’t tell you which case you’re in.
No devices were foundWhen plain nvidia-smi prints it with no -i at all, the driver is up but can’t see a card. Work through the causes in order of how cheap they are to rule out:
- Look for a wrong
-iindex or UUID in the script that called it. - Confirm the VM or container was actually given the GPU.
- Read the kernel log for the GPU dropping off the bus.
Don’t try to hide or reorder GPUs with CUDA_VISIBLE_DEVICES either: nvidia-smi ignores it, and in the lab CUDA_VISIBLE_DEVICES="" nvidia-smi -L still listed GPU 0. Use -i with a UUID or bus ID instead.
NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver
Here the kernel module isn’t loaded at all, so there’s nothing for nvidia-smi to talk to. This wasn’t reproduced for this guide, because a rented container can’t unload the host’s driver. The common causes reported on the NVIDIA developer forum are a kernel update that DKMS didn’t rebuild the module for, and Secure Boot rejecting an unsigned module. Reinstalling the driver cleanly fixes both, and the steps differ by distro: follow the Ubuntu driver and CUDA install or the Fedora driver guide, both of which cover Secure Boot key enrollment.
Keep this nvidia-smi command reference
Every flag below was run on the lab card (-i as -i 1), including the power limit change that the container refused. Keep the table next to your terminal, since it covers what you need for day-to-day GPU checks.
| Command | What it does |
|---|---|
nvidia-smi | One snapshot of every GPU plus the process table |
nvidia-smi -L | List GPUs with UUIDs |
nvidia-smi --version | Driver, NVML and CUDA versions |
watch -n 1 nvidia-smi | Redraw the table in place every second |
nvidia-smi -l 1 | Print a new table every second (default 5 with no value) |
--query-gpu=FIELDS --format=csv | Chosen fields as CSV; add noheader,nounits for scripts |
-l 1 -f gpu-log.csv | Loop a query and write it to a file (overwrites) |
-lms 200 | Loop interval in milliseconds |
--query-compute-apps=pid,process_name,used_memory --format=csv | VRAM per process |
nvidia-smi dmon -s pucm -o T | Scrolling device stats with timestamps |
nvidia-smi pmon -s um | Scrolling per-process utilization and memory |
nvidia-smi -q -d POWER,TEMPERATURE,PERFORMANCE | Detailed power, temperature and clock event reasons |
--query-gpu=clocks_event_reasons.active | Throttling bitmask; all zeros means no limit is active |
nvidia-smi -i 0 | Target one GPU by index, UUID or PCI bus ID |
nvidia-smi -pl 250 | Set the power limit in watts (root on the host) |
nvidia-smi --help-query-gpu | Every queryable field with a description |
The lab card was a vast.ai (paid link) rental at about $0.23 an hour, which is enough to try every command here on a GPU you don’t own. More command references for other tools live on the cheat sheets page.
