NVFP4 made Qwen3-8B 30% faster than FP8 on an RTX 5090 and cost it 1.5 points of GSM8K. Against BF16 the same 4-bit checkpoint was 2.1 times faster at 32 concurrent requests and 2.2 times faster for a single user. FP8 sat within the error bars of BF16 on accuracy. Those numbers come from serving all three precisions of one model in TensorRT-LLM on one consumer Blackwell card. The config, the client and the 1,319-question benchmark didn’t change between runs.
Published NVFP4 vs FP8 results almost all come from B200 data-center parts. Our RTX PRO 6000 vs RTX 5090 comparison covers the workstation cards, but not 4-bit floats. This test covers NVFP4 quantization on GeForce Blackwell (SM120): file size, weight memory, KV cache capacity, single-request speed, batched throughput with trtllm-bench, and GSM8K accuracy through lm-evaluation-harness. It also covers the one warning these NVIDIA NVFP4 checkpoints print at load time. Every number below was run on an RTX 5090 with the 595 driver and TensorRT-LLM 1.3.0rc29 in October 2026.
NVFP4 vs FP8 vs BF16 results on the RTX 5090
All three checkpoints are Qwen3-8B. BF16 is the original Qwen/Qwen3-8B. FP8 and NVFP4 are nvidia/Qwen3-8B-FP8 and nvidia/Qwen3-8B-NVFP4, which NVIDIA quantized with ModelOpt and publishes ungated. Every row ran on the same server config, with 32 slots of 8,192 tokens and 80% of free memory given to KV cache. The two request-speed rows come from the streaming client used in the Ollama, vLLM and llama.cpp comparison, which sends a 472-token prompt and asks for exactly 256 tokens. The trtllm-bench rows come from the commands shown below.
| Measure | BF16 | FP8 | NVFP4 |
|---|---|---|---|
| Checkpoint on disk | 16.4 GB | 9.5 GB | 6.4 GB |
| Weights in VRAM | 15.31 GiB | 8.84 GiB | 6.02 GiB |
| KV cache dtype (from checkpoint) | BF16 | FP8 | FP8 |
| KV cache capacity | 78,752 tokens | 232,928 tokens | 265,280 tokens |
| Single request, decode tokens/s | 99.8 | 170.0 | 222.7 |
| 32 concurrent requests, tokens/s | 2,514 | 4,118 | 5,340 |
| trtllm-bench throughput, output tokens/s | 1,809 | 3,310 | 4,494 |
| trtllm-bench TTFT average, 32 at once | 806 ms | 354 ms | 221 ms |
| GSM8K strict-match | 86.66% | 86.43% | 84.91% |
| GSM8K flexible-extract | 86.96% | 86.66% | 84.69% |
| GSM8K wall time (1,319 questions) | 232 s | 135 s | 131 s |
The GSM8K standard error was 0.93 to 0.99 points for every run. FP8 is inside that band of BF16 on both metrics. NVFP4 sits 1.5 points below FP8 on strict-match and 2.0 on flexible-extract. That is roughly 1.1 to 1.4 times the combined error. Treat it as a small, consistent drop, not a proven large one.
What NVFP4 stores, and why the RTX 5090 can run it
NVFP4 stores each weight as a 4-bit float (E2M1: one sign bit, two exponent bits, one mantissa bit). Every block of 16 values shares an FP8 E4M3 scale, and each tensor gets one FP32 scale on top. MXFP4, the other 4-bit float format, uses 32-value blocks with a power-of-two scale. The smaller blocks and finer FP8 scales are what NVIDIA credits for the accuracy of NVFP4. The NVIDIA NVFP4 write-up covers the format in detail.
Blackwell tensor cores multiply FP4 natively, and that includes the GeForce parts. The quantization support matrix for the current TensorRT-LLM release marks NVFP4, FP8 per-tensor and FP8 KV cache as supported on SM120. The NVIDIA checkpoint keeps lm_head in higher precision and stores its KV cache quantization as FP8, which the server log confirms at load:
[10/12/2026-09:27:45] [TRT-LLM] [I] [llmapi] Setting quant_algo=NVFP4 from HF quant config.
[10/12/2026-09:27:45] [TRT-LLM] [I] [llmapi] Setting kv_cache_quant_algo=FP8 from HF quant config.
[10/12/2026-09:27:45] [TRT-LLM] [I] [llmapi] Setting group_size=16 from HF quant config.
[10/12/2026-09:27:45] [TRT-LLM] [I] [llmapi] Setting exclude_modules=['lm_head'] from HF quant config.That FP8 KV line matters for reading the table. The FP8 checkpoint also sets kv_cache_quant_algo=FP8, while BF16 keeps its KV cache in BF16. Part of the throughput gap at 32 requests comes from KV cache precision, and the weight format accounts for the rest. The single-request numbers isolate the weight format better, because one request never comes close to filling any of the three KV pools.
Download the three Qwen3-8B checkpoints
Everything ran inside the TensorRT-LLM container, started as in the TensorRT-LLM Docker install guide with the host directory ~/trtllm mounted at /workspace. The image already ships the hf CLI. Set a variable for the model directory in the container shell:
export M=/root/modelsPull the checkpoints into one directory. None of the three repos is gated:
for m in nvidia/Qwen3-8B-FP8 nvidia/Qwen3-8B-NVFP4 Qwen/Qwen3-8B; do
hf download $m --local-dir $M/$(basename $m)
done
du -sh $M/*The 4-bit checkpoint is under 40% of the BF16 one:
16G /root/models/Qwen3-8B
8.9G /root/models/Qwen3-8B-FP8
6.0G /root/models/Qwen3-8B-NVFP4/root/models lives in the container filesystem, which is what we used. Removing the container deletes the 31 GB of checkpoints, so download to /workspace/models instead if you want them to survive. These are pre-quantized checkpoints. The ModelOpt hf_ptq.py script can produce your own NVFP4 checkpoint from a supported BF16 model. Most readers will deploy the published ones, so those are what we measured.
Measure throughput with trtllm-bench
Each precision used the same config file, the one from the install guide. It caps the batch at 32 and the context at 8,192 tokens. If you skipped that guide, create it on the host:
vim ~/trtllm/config.yamlThese are the exact contents used for every run:
max_batch_size: 32
max_num_tokens: 8192
max_seq_len: 8192
kv_cache_config:
free_gpu_memory_fraction: 0.80
enable_block_reuse: true
cuda_graph_config:
enable_padding: true
max_batch_size: 32Stop any running trtllm-serve with Ctrl+C first, because the benchmark needs the GPU memory the server reserved for KV cache. Generate one synthetic dataset and reuse it for all three, so the token counts are identical. This one is 256 requests of 512 input and 256 output tokens:
trtllm-bench --model nvidia/Qwen3-8B-FP8 prepare-dataset --output /workspace/ds_512_256.txt \
token-norm-dist --num-requests 256 --input-mean 512 --input-stdev 0 --output-mean 256 --output-stdev 0The throughput run sends 32 requests at a time. --model names the tokenizer family and --model_path points at the local checkpoint, so swap only the path between runs:
trtllm-bench --model Qwen/Qwen3-8B --model_path $M/Qwen3-8B-NVFP4 throughput \
--dataset /workspace/ds_512_256.txt --config /workspace/config.yaml \
--concurrency 32 --streaming --report_json /workspace/tput-NVFP4.jsonThe NVFP4 report, trimmed to the summary lines:
Request Throughput (req/sec): 17.5548
Total Output Throughput (tokens/sec): 4494.0291
Average request latency (ms): 1820.0336
Average time-to-first-token [TTFT] (ms): 221.2054
Per User Output Speed (tps/user): 159.6575
[TPOT] P50 : 6.4080The same command with latency in place of throughput sends one request at a time. Use 16 requests, since the result settles quickly:
trtllm-bench --model Qwen/Qwen3-8B --model_path $M/Qwen3-8B-NVFP4 latency \
--dataset /workspace/ds_512_256.txt --config /workspace/config.yaml \
--num_requests 16 --report_json /workspace/lat-NVFP4.jsonDecode on one request is memory-bandwidth bound, so speed follows the bytes read per token. In the trtllm-bench latency runs, cutting the weight bytes by 42% from BF16 to FP8 took per-user speed from 99.8 to 169.9 tokens per second. NVFP4 took it to 223.1, a smaller 31% step. The weights only shrank by 32% on that step (8.84 to 6.02 GiB), not by half, because the NVIDIA checkpoint keeps the embeddings and lm_head in BF16. That is about 2.3 GiB that never gets smaller, and lm_head is read for every token.
Check NVFP4 accuracy with GSM8K
Speed numbers mean nothing if the 4-bit model returns garbage. An open TensorRT-LLM issue reports wrong NVFP4 GEMM output on some SM120 machines, filed against this same Qwen3-8B-NVFP4 checkpoint on an RTX 5060 Ti. Run an accuracy check on your own card before trusting any benchmark. We used the same harness and settings as the GGUF vs AWQ vs GPTQ vs FP8 test: lm-evaluation-harness against the running server, 5-shot GSM8K, all 1,319 questions.
lm-eval has its own dependency tree. Give it a separate virtual environment inside the container so it cannot disturb the TensorRT-LLM packages:
pip install -q uv && uv venv -q -p 3.12 /root/lme
uv pip install -q -p /root/lme/bin/python "lm_eval[api]==0.4.13" transformers #https://github.com/EleutherAI/lm-evaluation-harness/releasesStart the server on the checkpoint under test. Stop it with Ctrl+C and start it again on the next checkpoint between runs:
trtllm-serve $M/Qwen3-8B-NVFP4 --host 0.0.0.0 --port 8000 --config /workspace/config.yamlThe server holds the terminal, so open a second shell in the container from the host:
docker exec -it trtllm bashThe new shell does not inherit the variable, so set it again and point lm-eval at the completions endpoint. The tokenizer comes from the BF16 directory so all three runs tokenize the few-shot prompts the same way:
export M=/root/models
/root/lme/bin/lm_eval --model local-completions \
--model_args model=$M/Qwen3-8B-NVFP4,base_url=http://127.0.0.1:8000/v1/completions,num_concurrent=16,tokenizer=$M/Qwen3-8B,tokenized_requests=False,max_retries=3 \
--tasks gsm8k --batch_size 1 --output_path /workspace/lmeval-NVFP4The NVFP4 run finished in just over two minutes:
|Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k| 3|flexible-extract| 5|exact_match|↑ |0.8469|± |0.0099|
| | |strict-match | 5|exact_match|↑ |0.8491|± |0.0099|84.9% strict-match is a model that works. The garbage-output bug in that issue produces repeated junk tokens, which would score near zero, not two points down. FP8 scored 86.43% and BF16 86.66%, so on this card FP8 is free accuracy-wise and NVFP4 is not.
GSM8K is one benchmark. Multi-step arithmetic is sensitive to quantization, which makes it a decent canary. It does not tell you how a 4-bit model handles long-context retrieval or code. If your workload is one of those, run the matching lm-eval task the same way before you switch.
Warning: Inline ‘config.json.quantization_config’ diverges from ‘hf_quant_config.json’
Every load of nvidia/Qwen3-8B-NVFP4 printed this warning, five times per server start and eight times per trtllm-bench run:
[10/12/2026-09:27:45] [TRT-LLM] [W] [quantize] Inline 'config.json.quantization_config' diverges from 'hf_quant_config.json':
- group_size: file=16, inline=NoneThe checkpoint carries its quantization settings twice. hf_quant_config.json has a top-level group_size: 16. The copy inside config.json nests the same 16 under its per-group settings and has no top-level key, which is the field TensorRT-LLM compares. TensorRT-LLM reads the dedicated file, logs Setting group_size=16 from HF quant config on the next line, and loads correctly. Nothing needs fixing, and the GSM8K score above came from a server that printed it.
When to stay on FP8
NVFP4 pays off when the GPU is the bottleneck: a 32 GB card serving many users, or a model that only fits in 4 bits. On the 5090 it bought 30% more throughput and 31% more single-user speed over FP8. It also cut 2.8 GiB of weights, which grew the KV pool by 2.2 GiB (about 32,000 tokens). That mattered less here, because the FP8 checkpoint already had over 230,000 tokens of room.
FP8 is the better default when accuracy is the product. It matched BF16 on GSM8K within the noise and still ran 1.7 times faster than BF16 for one user. Stay on FP8 if any of these apply:
- The workload is math, code or structured extraction, where a 1.5 point drop on GSM8K shows up as wrong answers users notice.
- The model already fits in FP8 with enough KV cache for your concurrency. An 8B model on a 32 GB card has KV room to spare.
- You run GPUs older than Blackwell. Hopper and Ada have FP8 tensor cores but no native FP4 math, and TensorRT-LLM lists no NVFP4 support for them.
- You cannot run an accuracy check on your own hardware first, given the open SM120 NVFP4 GEMM issue.
For bigger models the trade flips. A 27B model does not fit a 32 GB card in FP8 with any useful KV cache. NVFP4 becomes the only way to run it on one 5090 at all. Our GPU guide for local LLMs sizes cards by the model and precision you plan to run. Serving with vLLM instead? The vLLM install guide covers that server, which also loads ModelOpt NVFP4 checkpoints. The Ollama, vLLM and llama.cpp comparison shows what each engine does with a 4-bit model on this card.
