FP8 cost Qwen3.5-9B nothing on GSM8K and made it 1.76 times faster than BF16. The two 4-bit vLLM formats, AWQ and GPTQ, were faster again but lost six to twelve points of GSM8K strict-match accuracy. That is the short version of a GGUF vs AWQ vs GPTQ vs FP8 test we ran on one 24GB card, with every checkpoint quantized from the same BF16 weights on the same box.
Most comparisons of these formats quote numbers from someone else’s H100 or A100. This one measures file size, VRAM used by the weights, single-request latency, batched throughput, KV cache capacity, GSM8K accuracy and, for GGUF, KL divergence from the BF16 model across eight llama.cpp quant levels. It also covers the errors that stop BF16 from starting in vLLM on a 24GB GPU, and why a 4-bit AWQ checkpoint is about 35% bigger than a 4-bit GGUF of the same model. Everything ran on an NVIDIA A10G 24GB in September 2026, using vLLM 0.30.0, llm-compressor 0.14.0 and a llama.cpp build from 29 September.
GGUF vs AWQ vs GPTQ vs FP8 results at a glance
The vLLM formats and the llama.cpp formats are measured with each runtime’s own benchmark tool, so compare speed within a table, not across the two (the Ollama vs vLLM vs llama.cpp comparison covers the engines themselves). GSM8K accuracy uses the same lm-evaluation-harness task (5-shot, all 1,319 questions) for every format.
| vLLM format | Weights in VRAM | Latency, 512 in / 128 out | Throughput, 200 requests (1,024 in / 128 out) | KV cache | GSM8K strict / flexible |
|---|---|---|---|---|---|
| BF16 | 16.8 GiB | 4.68 s | 117 tok/s | 17,873 tokens | 87.5% / 87.2% |
| FP8 dynamic | 10.38 GiB | 2.66 s | 338 tok/s | 166,074 tokens | 87.5% / 87.0% |
| GPTQ W4A16 | 7.27 GiB | 1.83 s | 363 tok/s | 238,312 tokens | 81.8% / 80.8% |
| AWQ W4A16 | 7.29 GiB | 1.84 s | 363 tok/s | 237,568 tokens | 75.7% / 82.0% |
| GGUF (llama.cpp) | Size | Prompt, pp512 | Generation, tg128 | Mean KLD vs BF16 | Same top token | GSM8K strict / flexible |
|---|---|---|---|---|---|---|
| BF16 | 17.13 GiB | 2,822 tok/s | 30.5 tok/s | 0 (reference) | 100% | not run |
| Q8_0 | 9.10 GiB | 3,266 tok/s | 53.2 tok/s | 0.0017 | 98.5% | 87.6% / 87.2% |
| Q6_K | 7.03 GiB | 3,011 tok/s | 64.8 tok/s | 0.0052 | 97.2% | not run |
| Q5_K_M | 6.18 GiB | 3,166 tok/s | 72.9 tok/s | 0.0263 | 93.9% | not run |
| Q4_K_M | 5.37 GiB | 3,254 tok/s | 81.6 tok/s | 0.0340 | 92.5% | 84.8% / 84.4% |
| IQ4_XS | 4.95 GiB | 3,333 tok/s | 89.2 tok/s | 0.0264 | 93.0% | 86.4% / 86.0% |
| Q3_K_M | 4.40 GiB | 3,110 tok/s | 88.3 tok/s | 0.0798 | 87.8% | 84.2% / 84.4% |
| Q2_K | 3.64 GiB | 2,729 tok/s | 103.6 tok/s | 0.2279 | 79.7% | 76.7% / 76.9% |
For a 24GB card the picks fall out of those numbers. Serving several users through vLLM, run FP8: it matches BF16 on accuracy and still leaves room for 166K tokens of KV cache. On a single-user llama.cpp or Ollama setup, IQ4_XS beats Q4_K_M on KL divergence, top-token agreement and GSM8K while being smaller and faster. Drop to AWQ or GPTQ only when the extra concurrency or the 3 GiB of VRAM matters more than six to twelve points of GSM8K accuracy.
How we tested
One AWS g5.2xlarge in eu-west-1: an NVIDIA A10G with 24GB (22.06 GiB usable to CUDA), 8 vCPUs and 32GB of RAM, on the AWS Deep Learning Base AMI with Ubuntu 24.04 and the 595 driver. The A10G is an Ampere card (compute capability 8.6), the same generation as the RTX 3090 and A5000, with about 600 GB/s of memory bandwidth against the 3090’s 936 GB/s. Treat the numbers as a relative ranking of formats; a faster card moves every row up.
The model is Qwen3.5-9B (Apache-2.0). It is a hybrid: 24 of its 32 layers are Gated DeltaNet linear attention and only 8 are full attention with 4 KV heads, which makes its KV cache small per token and changes how it behaves in vLLM (more on that below). Every quantized version was produced on the box from the same BF16 weights, so no row depends on someone else’s recipe.
Each metric has one method:
- Weights in VRAM: vLLM’s own “Model loading took X GiB” log line; for GGUF, the model size llama-bench reports.
- Latency:
vllm bench latency, one request of 512 prompt tokens and 128 output tokens, 2 warmup and 5 measured iterations, average. - Throughput:
vllm bench throughput, 200 requests, output tokens per second. The benchmark ignored our--input-len 512 --output-len 256and used its random-dataset defaults of 1,024 prompt tokens and 128 output tokens, so that is the workload behind every throughput row. - KV cache: the “GPU KV cache size” line vLLM prints at startup with
--gpu-memory-utilization 0.90,--max-num-seqs 32,--max-model-len 4096, taken from the latency run. The same flags gave different figures on other starts (BF16 ranged from 14,043 to 35,986 tokens), so read them as approximate. - GGUF speed:
llama-bench -ngl 99 -fa 1 -p 512 -n 128 -r 3, all layers on the GPU. - KL divergence and top token:
llama-perplexityon the wikitext-2 test set, 150 chunks of 512 tokens, each quant scored against the BF16 GGUF’s logits. - GSM8K: lm-evaluation-harness, 5-shot, greedy, all 1,319 questions. vLLM formats through its
vllmbackend; GGUF throughllama-serverand thelocal-completionsbackend. Strict-match needs the#### answerformat, flexible-extract takes the last number in the reply.
The same vLLM flags applied to every format, including BF16, so the throughput rows are comparable. The commands in the sections below are the ones that produced every row.
GGUF quantization quality from Q8_0 to Q2_K
KL divergence measures how far a quant’s whole probability distribution drifts from the BF16 model on every token, which is a much finer signal than perplexity. The “same top token” column is simpler: the share of positions where the quant would pick the same next token as BF16.
| GGUF | Perplexity | Mean KLD | 99th pct KLD | Same top token | RMS Δp | Quantize time |
|---|---|---|---|---|---|---|
| BF16 | 7.738 | 0 | 0 | 100% | 0 | 201 s (convert) |
| Q8_0 | 7.750 | 0.0017 | 0.0085 | 98.5% | 1.3% | 56 s |
| Q6_K | 7.747 | 0.0052 | 0.0227 | 97.2% | 2.0% | 150 s |
| Q5_K_M | 7.660 | 0.0263 | 0.2154 | 93.9% | 4.3% | 222 s |
| IQ4_XS | 7.857 | 0.0264 | 0.2155 | 93.0% | 4.4% | 476 s |
| Q4_K_M | 7.755 | 0.0340 | 0.2785 | 92.5% | 5.0% | 223 s |
| Q4_K_M, no imatrix | 7.763 | 0.0572 | 0.4619 | 89.9% | 6.4% | 149 s |
| Q3_K_M | 8.194 | 0.0798 | 0.6760 | 87.8% | 7.9% | 141 s |
| Q2_K | 8.914 | 0.2279 | 1.9649 | 79.7% | 13.5% | 218 s |
Three things stand out. IQ4_XS lands on top of Q5_K_M for KL divergence while being 1.2 GiB smaller, and it beats Q4_K_M on everything except raw perplexity (7.857 against 7.755). Q8_0 and Q6_K are effectively lossless. Below Q3_K_M the curve breaks: Q2_K picks a different token from BF16 one time in five and its 99th percentile KLD is seven times Q4_K_M’s.
The importance matrix is the cheapest quality you will ever buy. The same Q4_K_M built without one has 68% more KL divergence and loses 2.6 points of top-token agreement, at an identical 5.37 GiB and an identical 81.6 tok/s. It cost 302 seconds to compute on the GPU and 75 extra seconds of quantizing. Every quant in the table except the “no imatrix” row used it.
Generation speed follows file size almost linearly because decoding is memory-bandwidth bound: Q4_K_M generates 2.7 times faster than BF16. Prompt processing barely moves (2,729 to 3,333 tok/s across all nine files) since it is compute bound, and Q2_K is actually the slowest at it because its dequantization costs more math. If your workload is long prompts with short answers, such as RAG or classification, a smaller quant buys you far less than the tg128 column suggests. The llama.cpp setup guide covers building the CUDA binaries used here.
AWQ vs GPTQ in vLLM
On speed they are the same format. Both are W4A16 with group size 128, both load through vLLM’s MarlinLinearKernel for CompressedTensorsWNA16, and they finished within 1% of each other on latency (1.83 s vs 1.84 s), throughput (363 tok/s each) and KV capacity. Choosing between them is purely about accuracy and how long you are willing to wait for calibration.
The GPTQ engine log shows the kernel, the weight footprint and the KV headroom a 4-bit checkpoint leaves on a 24GB card:
Calibration is where they differ. With the same 256 samples from the perfectblend dataset at 2,048 tokens, GPTQ’s one-shot pass took 433 seconds and peaked at 4.7 GiB of GPU memory. AWQ’s grid search over scales took 1,661 seconds (3.8 times longer) and peaked at 13.2 GiB. FP8 dynamic needs no calibration data at all and quantized in 3 seconds.
Accuracy is where the split gets interesting. GPTQ scored 81.8% strict and 80.8% flexible on GSM8K. AWQ scored higher on flexible-extract (82.0%) but fell to 75.7% on strict-match, which means it solved roughly as many problems but broke the expected #### answer format more often. That kind of formatting drift is a real 4-bit side effect: if your application parses structured output, test the exact format you depend on, not just a headline benchmark.
The GGUF side did better at the same nominal bit width. Q4_K_M scored 84.8% strict on GSM8K and IQ4_XS 86.4%, both ahead of GPTQ’s 81.8% and within about one to three points of BF16. The llama.cpp quant mixes are not a flat 4 bits. Our Q4_K_M quantize log shows the output head, half of the ffn_down tensors (16 of 32), 12 attn_qkv and 4 attn_v tensors stored as Q6_K, while the llm-compressor W4A16 recipe applies 4 bits to every Linear layer it touches. On this model, that difference was worth more accuracy than the choice between AWQ and GPTQ.
Both use the stock modifier settings from the llm-compressor examples (symmetric W4A16 for GPTQ, asymmetric W4A16 with duo_scaling="both" for AWQ), with calibration unified at 256 samples of 2,048 tokens for both. More calibration data, a domain-matched dataset or act-order may close part of the gap; we did not tune them.
FP8 on an Ampere card without FP8 hardware
The A10G has no FP8 tensor cores, yet the FP8_DYNAMIC checkpoint loaded and ran in vLLM without a flag or a warning. On Turing and Ampere cards vLLM runs FP8 checkpoints as weight-only W8A16 through its FP8 Marlin kernels, so the weights stay 8-bit in memory and the math runs in 16-bit (per the vLLM FP8 docs; true FP8 compute needs compute capability 8.9 or newer). It cut single-request latency from 4.68 s to 2.66 s, matched BF16 on GSM8K strict-match to the decimal (87.49%) and freed enough memory for 166,074 tokens of KV cache against BF16’s 17,873. That last number is what matters for serving: at a 4,096-token context, BF16 fits 4.4 concurrent requests on this card and FP8 fits 40.
The throughput run shows the effect. BF16 managed 117 output tokens per second over 200 requests because its tiny KV cache kept only a handful of sequences in flight. FP8 ran at the full 32-sequence cap and delivered 338 tok/s, 2.9 times more, while the 4-bit formats added only another 7% on top. Once the KV cache stops being the limit, batched decoding is bound by compute, and the smaller weights of 4-bit stop paying off. If you are sizing a card for a team, the VRAM sizing guide walks through the same trade.
Why a 4-bit AWQ checkpoint is bigger than a 4-bit GGUF
GPTQ and AWQ put 7.27 and 7.29 GiB of weights on the GPU. Q4_K_M, the same nominal 4 bits, is 5.37 GiB. The gap is almost entirely two tensors. Qwen3.5 has a 248,320-token vocabulary, so the input embedding and the output head are each 4096 × 248,320 values, about 1.9 GiB apiece in BF16. The llm-compressor recipe leaves both at BF16 (its ignore list covers embed_tokens and lm_head), plus the small DeltaNet in_proj_a and in_proj_b projections.
llama.cpp quantizes both. The Q4_K_M quantize log shows the output head stored as Q6_K (1,940 MiB down to 796 MiB) and the embedding as Q4_K (down to 546 MiB). The imatrix lines between them are trimmed:
[ 1/ 442] output.weight - [ 4096, 248320, 1, 1], type = bf16,
====== llama_model_quantize_impl: did not find weights for output.weight
converting to q6_K .. size = 1940.00 MiB -> 795.70 MiB
[ 3/ 442] token_embd.weight - [ 4096, 248320, 1, 1], type = bf16,
====== llama_model_quantize_impl: did not find weights for token_embd.weight
converting to q4_K .. size = 1940.00 MiB -> 545.62 MiBThe on-disk folders are larger again (8.5 GB for GPTQ and AWQ, 12 GB for FP8) because llm-compressor also saves the model’s vision encoder unquantized, which vLLM skips when started with --language-model-only. GGUF keeps vision in a separate mmproj file. Our file listing from the lab box:
Errors that stop BF16 from starting in vLLM on 24GB
BF16 Qwen3.5-9B does not start in vLLM on a 24GB card with default settings. It failed three times before it ran, and the same flags were then applied to every format for a fair comparison.
Error: “torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 4.12 GiB”
The first run loaded 17.66 GiB of weights, vision tower included, then failed to allocate 4.12 GiB for the cache it builds during CUDA graph memory profiling, with 3.70 GiB free. The model is multimodal, so vLLM also keeps the vision encoder’s weights and its encoder budget resident. For text-only work, drop them:
--language-model-onlyThat dropped the weights to 16.8 GiB and got past profiling, only to fail on a 512 MiB allocation during CUDA graph capture inside the Gated DeltaNet attention kernel. At vLLM’s default concurrency, graph capture reserves linear-attention state for more sequences than the card has memory left for.
Error: “ValueError: max_num_seqs (64) exceeds available Mamba cache blocks (43)”
Lowering the batch to 64 produced a much more honest error:
ValueError: max_num_seqs (64) exceeds available Mamba cache blocks (43). Each decode sequence requires one Mamba cache block, so CUDA graph capture cannot proceed. Please lower max_num_seqs to at most 43 or increase gpu_memory_utilization.Every sequence vLLM decodes needs its own block of DeltaNet state, and with 16.8 GiB of BF16 weights only 43 of them fit. The fix is to cap concurrency below that number. The quantized formats have room for far more, but we kept the same cap for all four so the throughput numbers compare like for like:
--language-model-only --max-num-seqs 32 --max-model-len 4096 --gpu-memory-utilization 0.90One more error came from llm-compressor rather than vLLM. Loading the Qwen3.5 processor failed with Qwen3VLVideoProcessor requires the Torchvision library but it was not found in your environment, because llmcompressor does not pull in torchvision. Installing torchvision into the same environment fixed it.
Quantize Qwen3.5-9B to AWQ, GPTQ and FP8 with llm-compressor
AutoAWQ and AutoGPTQ are both archived now. AutoGPTQ’s README recommends GPTQModel, and AutoAWQ’s points to llm-compressor, the vLLM project’s quantization library, which produces AWQ, GPTQ and FP8 checkpoints in the compressed-tensors format vLLM loads directly. Install it in its own environment with torchvision:
uv venv ~/llmc-env --python 3.12
VIRTUAL_ENV=~/llmc-env uv pip install "llmcompressor==0.14.0" datasets #https://github.com/vllm-project/llm-compressor/releases
VIRTUAL_ENV=~/llmc-env uv pip install torchvision pillowThe pin matters here: the current examples import the AWQ modifier from llmcompressor.modifiers.transform.awq, and snippets written for older releases use a different path. uv resolved torchvision to the build matching llm-compressor’s torch. vLLM and lm-evaluation-harness go in a second environment, separate from llm-compressor so their torch pins never fight, and the hf CLI comes from that one:
uv venv ~/vllm-env --python 3.12
VIRTUAL_ENV=~/vllm-env uv pip install "vllm==0.30.0" "lm_eval[vllm,api]" huggingface_hub hf_transfer #https://github.com/vllm-project/vllm/releasesDownload the BF16 weights once. Every format below starts from this copy:
~/vllm-env/bin/hf download Qwen/Qwen3.5-9B --local-dir ~/models/Qwen3.5-9BCreate the quantization script:
vim ~/quantize.pyThis is the exact script that produced the three checkpoints in the tables. The ignore list keeps the vision tower, embeddings, output head and the two small DeltaNet projections in BF16, following the Qwen example that ships with llm-compressor:
import sys, time, torch
from transformers import AutoProcessor, Qwen3_5ForConditionalGeneration
from llmcompressor import oneshot
from llmcompressor.modifiers.gptq import GPTQModifier
from llmcompressor.modifiers.quantization import QuantizationModifier
from llmcompressor.modifiers.transform.awq import AWQModifier
from llmcompressor.utils import load_context
MODEL_ID = "/home/ubuntu/models/Qwen3.5-9B"
MODE = sys.argv[1] # gptq | awq | fp8
IGNORE = [
"re:visual.*",
"re:model.visual.*",
r"re:.*lm_head",
"re:.*embed_tokens$",
r"re:.*linear_attn\.in_proj_a$",
r"re:.*linear_attn\.in_proj_b$",
]
with load_context(Qwen3_5ForConditionalGeneration):
model = Qwen3_5ForConditionalGeneration.from_pretrained(MODEL_ID, dtype="auto")
processor = AutoProcessor.from_pretrained(MODEL_ID)
if MODE == "gptq":
recipe = GPTQModifier(targets="Linear", scheme="W4A16", ignore=IGNORE)
elif MODE == "awq":
recipe = [
AWQModifier(duo_scaling="both"),
QuantizationModifier(targets="Linear", scheme="W4A16_ASYM", ignore=IGNORE),
]
elif MODE == "fp8":
recipe = QuantizationModifier(targets="Linear", scheme="FP8_DYNAMIC", ignore=IGNORE)
t0 = time.time()
if MODE == "fp8":
oneshot(model=model, recipe=recipe)
else:
oneshot(
model=model,
processor=processor,
recipe=recipe,
dataset="perfectblend",
splits="train[:512]",
max_seq_length=2048,
num_calibration_samples=256,
)
elapsed = time.time() - t0
SAVE_DIR = f"/home/ubuntu/models/Qwen3.5-9B-{MODE.upper()}"
model.save_pretrained(SAVE_DIR, save_compressed=True)
processor.save_pretrained(SAVE_DIR)
print(f"RESULT mode={MODE} oneshot_seconds={elapsed:.0f} peak_gpu_gib={torch.cuda.max_memory_allocated()/2**30:.1f}")Run it once per format. FP8 finishes in under two minutes including load and save, GPTQ in about nine and AWQ in about half an hour on the A10G:
~/llmc-env/bin/python ~/quantize.py fp8
~/llmc-env/bin/python ~/quantize.py gptq
~/llmc-env/bin/python ~/quantize.py awqBenchmark the vLLM checkpoints yourself
Activate the vLLM environment created earlier:
source ~/vllm-env/bin/activateSet the model path and the shared flags once. These are the flags every row in the vLLM table ran with:
export P=~/models/Qwen3.5-9B-GPTQ
export COMMON="--language-model-only --max-num-seqs 32 --max-model-len 4096 --gpu-memory-utilization 0.90"Single-request latency, then batched throughput:
vllm bench latency --model $P $COMMON --input-len 512 --output-len 128 --batch-size 1 --num-iters-warmup 2 --num-iters 5
vllm bench throughput --model $P $COMMON --input-len 512 --output-len 256 --num-prompts 200The latency benchmark honours --input-len. The throughput benchmark does not: it defaults to the random dataset, and its own length flags win. It says so in a warning that is easy to miss in the startup noise:
UserWarning: Both --input-len and --random-input-len are specified. The random version (--random-input-len) will be preferred in this run.
UserWarning: Both --output-len and --random-output-len are specified. The random version (--random-output-len) will be preferred in this run.The run log then reports Sampling input_len from [1024, 1024] and output_len from [128, 128], with 204,800 prompt tokens and 25,600 output tokens in total. To benchmark a different shape, pass --random-input-len and --random-output-len instead.
GSM8K goes through lm-evaluation-harness, which starts its own vLLM engine with the same settings passed as model arguments:
lm_eval --model vllm --model_args pretrained=$P,language_model_only=True,max_num_seqs=32,max_model_len=4096,gpu_memory_utilization=0.90 --tasks gsm8k --batch_size autoA full GSM8K run took 10 to 14 minutes per format on the A10G. For GGUF, start llama-server with 16 parallel slots and point the local-completions backend at it; each quant took about 21 minutes that way:
~/llama.cpp/build/bin/llama-server -m ~/models/q35-IQ4_XS.gguf -ngl 99 -fa on -np 16 -c 65536 --port 8080 &
lm_eval --model local-completions --model_args model=q35,base_url=http://127.0.0.1:8080/v1/completions,num_concurrent=16,tokenizer=$HOME/models/Qwen3.5-9B,tokenized_requests=False,max_retries=3 --tasks gsm8k --batch_size 1Build GGUF quants with an importance matrix
The GGUF side needs a CUDA build of llama.cpp (see the NVIDIA driver and CUDA guide if the toolkit is not installed yet). The A10G is compute capability 8.6; set the architecture to match your card. The build tools and unzip (used later by the wikitext script) come first:
sudo apt-get update && sudo apt-get install -y cmake build-essential libcurl4-openssl-dev unzip
git clone --depth 1 https://github.com/ggml-org/llama.cpp.git ~/llama.cpp
cd ~/llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j 8Convert the Hugging Face checkpoint to a BF16 GGUF. The converter lives in the repo and needs its own Python requirements:
uv venv ~/gguf-env --python 3.12
VIRTUAL_ENV=~/gguf-env uv pip install -r ~/llama.cpp/requirements/requirements-convert_hf_to_gguf.txt
~/gguf-env/bin/python ~/llama.cpp/convert_hf_to_gguf.py ~/models/Qwen3.5-9B --outtype bf16 --outfile ~/models/q35-BF16.ggufCompute the importance matrix on the GPU. We used bartowski’s widely shared calibration text; any few hundred KB of varied text works:
curl -sL -o ~/calib.txt https://gist.githubusercontent.com/bartowski1182/eb213dccb3571f863da82e99418f81e8/raw/calibration_datav3.txt
~/llama.cpp/build/bin/llama-imatrix -m ~/models/q35-BF16.gguf -f ~/calib.txt -o ~/models/q35.imatrix.gguf -ngl 99Then quantize. Pass the imatrix to every level; this loop builds the whole ladder from the table:
for Q in Q8_0 Q6_K Q5_K_M Q4_K_M IQ4_XS Q3_K_M Q2_K; do
~/llama.cpp/build/bin/llama-quantize --imatrix ~/models/q35.imatrix.gguf ~/models/q35-BF16.gguf ~/models/q35-$Q.gguf $Q
doneBenchmark the new files with llama-bench using the same flags as the table. Our rerun for the two 4-bit candidates came back within noise of the first pass, with IQ4_XS about 9% faster at generation:
To score your own quants the same way, fetch wikitext-2 (the script unzips into the current directory, hence the cd ~), record the BF16 logits once and point each quant at them. The base file for 150 chunks of this model is 18 GB, so put it on a disk with room:
cd ~ && bash ~/llama.cpp/scripts/get-wikitext-2.sh
~/llama.cpp/build/bin/llama-perplexity -m ~/models/q35-BF16.gguf -f ~/wikitext-2-raw/wiki.test.raw -ngl 99 -c 512 --chunks 150 --kl-divergence-base ~/models/base.kld
~/llama.cpp/build/bin/llama-perplexity -m ~/models/q35-IQ4_XS.gguf -ngl 99 -c 512 --chunks 150 --kl-divergence-base ~/models/base.kld --kl-divergenceEach quant’s KL divergence pass took about three minutes of wall clock here (four for the first), and the GPU sat mostly idle while it streamed the 18 GB base file from disk. The llama-perplexity README documents the rest of the statistics it prints.
Formats we left out
NVFP4 is native W4A4 only on Blackwell GPUs; on an Ampere card vLLM falls back to a weight-only path, so it would not show what the format is for. EXL3 (exllamav3) needs its own runtime and cannot be served by vLLM or llama.cpp. HQQ has had no release since October 2025. GGUF inside vLLM is still marked highly experimental and has moved to an out-of-tree plugin, so GGUF was tested where people run it, in llama.cpp. Readers choosing hardware for any of these can start with the best GPU for local LLMs roundup.
Three quantization myths this benchmark breaks
“4-bit means a quarter of the size.” The 4-bit vLLM checkpoints put 7.27 to 7.29 GiB on the GPU, 43% of BF16, because the embeddings and output head stay in BF16. Only GGUF, which quantizes those too, gets near a third (5.37 GiB for Q4_K_M). Budget VRAM from the measured weight size, never from the bit count.
“Perplexity tells you whether a quant is good.” Q4_K_M’s perplexity is within 0.3% of BF16, yet it picks a different next token 7.5% of the time. Q5_K_M even scored a lower perplexity than BF16, which is noise, not an improvement. KL divergence and a task benchmark like GSM8K catch what perplexity hides.
“FP8 needs an H100.” It worked on an Ampere A10G with no FP8 hardware (as weight-only FP8), served requests 1.76 times faster than BF16 and lost nothing on GSM8K. For anyone serving a model of this size to more than one user on a 24GB card, FP8 is the first format to try, and 4-bit is the one to justify. The vLLM production install guide covers the service setup around the flags shown above.
