A long prompt costs more than compute time. It costs memory, and that memory is the KV cache. According to DeepSeek’s V4.1 announcement, its new architecture needs roughly one quarter of the previous generation’s HBM for the cache and one eighth of its SSD footprint. When a model lab brags about cache size, you know the cache has become the main constraint on long-context serving.
This KV cache tutorial walks through 13 steps in about 55 minutes. You will calculate how much cache any open-weight model needs from its config file, launch a baseline server, then shrink the cache with FP8 storage and prefix caching. Every change gets measured, so you finish with numbers from your own GPU instead of guesses. If you have never installed the engine, the earlier guide on how to set up vLLM covers the basics.
The timing is good. vLLM 0.32.0 was published by October 19, 2026, and the project’s own release headline calls it “low-precision KV cache everywhere.” DeepSeek-V4.1-Flash-1017 is the latest V4.1-Flash update, with a 1-million-token context window, according to third-party model listings. A million tokens of history sounds great until you do the arithmetic on your GPU. By the end of this guide you will do that arithmetic in one command.
Don't miss new tech stories on Google
Add Tech Insider once in the Google app and our stories appear in your news suggestions.
What a KV Cache Is and Why It Fills Your GPU
A transformer generates text one token at a time. For each new token, attention needs the key and value vectors of every earlier token. Recomputing them on every step would make generation painfully slow, so the engine stores them in GPU memory. That store is the KV cache. It grows by one entry per token per layer, and it stays until the request finishes.
Two things make the cache painful. First, it scales with context length, so a request carrying 100,000 tokens of history holds 100 times the cache of a 1,000-token chat. Second, it scales with concurrency. Ten simultaneous users each hold their own cache, and the GPU must fit all of them next to the model weights. When the pool runs out, vLLM queues new requests or evicts running ones and recomputes them later. Both outcomes hurt latency.
The per-token size follows a simple formula for standard multi-head and grouped-query attention models:
kv_bytes_per_token = 2 * num_layers * num_kv_heads * head_dim * bytes_per_elementThe leading 2 covers keys plus values. The last factor is where quantization pays off: BF16 uses 2 bytes per element and FP8 uses 1. Take an illustrative model with 36 layers, 8 KV heads and a head dimension of 128. That works out to 147,456 bytes per token at BF16, or 144 KiB. At FP8 it drops to 72 KiB. The table below applies the formula across common context lengths. These figures are arithmetic on the example config, not measurements of a specific product.
| Context per request | BF16 cache (144 KiB/token) | FP8 cache (72 KiB/token) | Saved |
|---|---|---|---|
| 8,192 tokens | 1.125 GiB | 0.5625 GiB | 0.5625 GiB |
| 32,768 tokens | 4.5 GiB | 2.25 GiB | 2.25 GiB |
| 131,072 tokens | 18 GiB | 9 GiB | 9 GiB |
| 262,144 tokens | 36 GiB | 18 GiB | 18 GiB |
| 1,048,576 tokens | 144 GiB | 72 GiB | 72 GiB |
Read the last row twice. A single 1M-token request on this example config needs 144 GiB of cache at BF16, before you load one weight. That is why 1M-context models target multi-GPU nodes, and why architectures that compress the cache matter so much. Not every model follows the simple formula, though. Models with multi-head latent attention store a compressed latent instead of full keys and values. Hybrid designs that mix attention with state-space layers keep a cache only in the attention layers. For those models the formula overestimates, which is fine for budgeting but wrong for bragging.
Prerequisites: Versions, Hardware and Accounts
You need a Linux machine with an NVIDIA GPU. The vLLM documentation lists compute capability 7.5 or higher, which covers a T4, an RTX 20-series card and everything newer, including A100, L4 and H100. The FP8 KV cache formats fp8 (an alias for fp8_e4m3) and fp8_e5m2 need CUDA 11.8 or later, according to the engine documentation. The current vLLM wheels target CUDA 13.x, so a recent driver is the real gate. AMD ROCm and Intel XPU paths exist, but this guide sticks to CUDA so every command is checkable.
| Component | Version or requirement | Why it matters |
|---|---|---|
| vLLM | 0.32.0 (published by Oct. 19, 2026) | Current release line, low-precision KV cache work |
| Python | 3.10 or newer, 3.12 recommended | Minimum from the vLLM install docs |
| CUDA | 13.x (13.0 and 13.1 listed as compatible) | Matches the cu130 wheel build |
| GPU | Compute capability 7.5 or higher | Required by current vLLM builds |
| GPU memory | 24 GB used in the examples | Fits an 8B model plus a small cache pool |
| Package manager | uv (latest) | Fast, repeatable environments |
| Model | Qwen/Qwen3-8B | Model used in vLLM’s own serving examples |
The examples use Qwen/Qwen3-8B because the vLLM documentation uses it in its serving examples, and because it fits on a 24 GB card with room left for a cache. That makes the cache the visible bottleneck, which is the point of the exercise. Swap in any model vLLM supports. If the repository is gated, create a Hugging Face account and export an access token as HF_TOKEN before you start.
Budget about 20 GB of free disk for the environment and weights. Close other GPU programs first. A browser with hardware acceleration or a running game can quietly hold a gigabyte or two of VRAM, and vLLM sizes its pool from whatever is free at startup. Check nvidia-smi before every launch in this guide, because a dirty baseline ruins every comparison you make later.
Steps 1 to 3: Install vLLM 0.32.0 and Check Your GPU
Why vLLM and not a desktop runner? Because vLLM prints its cache capacity at startup and exposes it as a metric, which makes the cache measurable. Desktop tools hide that detail on purpose. The trade-offs between the two are laid out in the vLLM vs Ollama comparison, and a later section here shows the Ollama equivalent for single-user machines.
Step 1: Create an isolated environment. Use uv so the install is repeatable. Install uv from its official instructions if you do not have it, then create a Python 3.12 virtual environment in a fresh folder.
mkdir kvlab
cd kvlab
uv venv --python 3.12
source .venv/bin/activateStep 2: Install vLLM at a pinned version. Pin the release so your measurements stay comparable if the project ships a new build next week. The cu130 backend flag selects the CUDA 13 build of PyTorch.
uv pip install vllm==0.32.0 --torch-backend=cu130
vllm download-kernelsThe second command surprises many readers. The current quickstart tells you to run vllm download-kernels after every install or upgrade, so precompiled FlashInfer kernels are fetched once instead of being downloaded and compiled during server startup. Skip it and your first launch can stall for minutes. That looks like a hang but is not.
Step 3: Verify the install and note the free memory. Confirm the version, confirm the CLI works, and write down the free VRAM. You will compare against this number later.
python -c "import vllm, torch; print(vllm.__version__, torch.cuda.is_available())"
vllm --help
nvidia-smi --query-gpu=name,memory.total,memory.used --format=csvExpected output looks like the block below. Your GPU name and memory will differ.
0.32.0 True
name, memory.total [MiB], memory.used [MiB]
NVIDIA GeForce RTX 4090, 24564 MiB, 412 MiBIf torch.cuda.is_available() prints False, stop here. The usual cause is a mismatch between the driver and the CUDA build you installed. Update the NVIDIA driver, or reinstall with the torch backend that matches your driver. Debugging a missing GPU after you launch a server is far more confusing than fixing it now.
Steps 4 and 5: Read the Model Config and Budget Your Cache
Step 4: Pull the four numbers from config.json. Every Hugging Face model ships a config.json. You need num_hidden_layers, num_attention_heads, num_key_value_heads and the head dimension. Some configs list head_dim directly. Others omit it, and you derive it as hidden_size divided by the attention head count. If num_key_value_heads is missing, the model uses plain multi-head attention and the KV head count equals the attention head count. Multimodal models often nest the language settings under text_config, so the script below checks for that.
import json
import sys
from huggingface_hub import hf_hub_download
repo = sys.argv[1]
path = hf_hub_download(repo, "config.json")
with open(path) as handle:
cfg = json.load(handle)
cfg = cfg.get("text_config", cfg)
layers = cfg["num_hidden_layers"]
heads = cfg["num_attention_heads"]
kv_heads = cfg.get("num_key_value_heads", heads)
head_dim = cfg.get("head_dim") or cfg["hidden_size"] // heads
print(f"layers={layers} kv_heads={kv_heads} head_dim={head_dim}")
for name, size in (("bf16", 2), ("fp8", 1)):
per_token = 2 * layers * kv_heads * head_dim * size
print(f"{name}: {per_token:,} bytes per token = {per_token / 1024:.1f} KiB")Save it as kv_bytes.py and run python kv_bytes.py Qwen/Qwen3-8B. For a config with 36 layers, 8 KV heads and a head dimension of 128, you should see this:
layers=36 kv_heads=8 head_dim=128
bf16: 147,456 bytes per token = 144.0 KiB
fp8: 73,728 bytes per token = 72.0 KiBStep 5: Turn bytes per token into a token budget. The cache pool is whatever is left after weights and overhead. vLLM takes a fraction of total GPU memory set by --gpu-memory-utilization, which defaults to 0.92 in the current engine arguments. Subtract the weights and a few GiB for activations and CUDA graphs, and the remainder becomes the pool. An 8B model at BF16 needs about 16 GB for weights (roughly 15.3 GiB), and 2 GiB of overhead is a reasonable starting guess.
import sys
total_gib = float(sys.argv[1])
util = float(sys.argv[2])
weights_gib = float(sys.argv[3])
overhead_gib = float(sys.argv[4])
kib_per_token = float(sys.argv[5])
pool_gib = total_gib * util - weights_gib - overhead_gib
tokens = int(pool_gib * 1024 * 1024 / kib_per_token)
print(f"cache pool: {pool_gib:.2f} GiB, about {tokens:,} tokens")
for ctx in (8192, 32768, 131072):
print(f" {ctx:>7,} tokens per request: {tokens / ctx:.2f} concurrent")Run python budget.py 24 0.92 15.3 2.0 144 for the BF16 case, then repeat with 72 for FP8. The arguments are total GiB, utilization, weights GiB, overhead GiB and KiB per token.
cache pool: 4.78 GiB, about 34,806 tokens
8,192 tokens per request: 4.25 concurrent
32,768 tokens per request: 1.06 concurrent
131,072 tokens per request: 0.27 concurrentThat last line is the whole story of long context on a small GPU. Before you run anything, write down two predictions: how many 131,072-token requests the BF16 cache can hold on your card, and how many tokens the FP8 cache will hold. Think about how many 32,768-token requests each run could serve at once. In step 7 you will check both predictions against what vLLM reports, and the gap between the two tells you how good your overhead guess was.
Steps 6 to 8: Launch a Baseline, Read the Log, Benchmark It
Step 6: Start a BF16 baseline server. Cap the context at 32,768 tokens so the engine does not try to reserve for the model’s full native window. Keep the default 0.92 utilization so the numbers match the prediction.
vllm serve Qwen/Qwen3-8B \
--max-model-len 32768 \
--gpu-memory-utilization 0.92 \
--port 8000Step 7: Read the three log lines that matter. Startup prints a lot. Scroll for the memory profiling summary. Three lines tell you what you need, and they appear after weights load and the engine profiles a dummy forward pass.
INFO Available KV cache memory: 4.78 GiB
INFO GPU KV cache size: 34,800 tokens
INFO Maximum concurrency for 32,768 tokens per request: 1.06xThe exact wording shifts between versions, and the token count is rounded to the engine’s block size, so expect small differences. If your numbers land within a few percent of the step 5 prediction, your overhead guess was good. If the real pool is much smaller, something else is holding memory, such as another process on the GPU or a model with a larger activation footprint than you assumed. Send one test request before moving on, to prove the server answers.
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "Qwen/Qwen3-8B", "messages": [{"role": "user", "content": "Say hello in five words."}], "max_tokens": 30}'The server listens on http://localhost:8000 by default and speaks the OpenAI API, so any OpenAI client works. That compatibility is also why other engines report the same fields differently. If you later compare against SGLang, the SGLang setup guide shows where its token pool figure appears.
Step 8: Benchmark the baseline under pressure. Use the built-in benchmark client, which talks to an OpenAI-compatible backend by default. Choose a load that the BF16 pool cannot serve in full. Each request here is 4,096 tokens in and 256 out, so 12 concurrent requests want more cache than the BF16 pool holds. Only some of the sequences fit at once and the rest wait. The FP8 pool should hold noticeably more of them at the same time.
vllm bench serve \
--model Qwen/Qwen3-8B \
--dataset-name random \
--random-input-len 4096 \
--random-output-len 256 \
--num-prompts 96 \
--max-concurrency 12 \
--save-result \
--result-filename bf16.jsonFlag names evolve, so run vllm bench serve --help if the CLI rejects one. The summary prints request throughput, output token throughput, time to first token (TTFT) and time per output token (TPOT). The layout looks like this, with placeholders where your own measurements go:
Successful requests: 96
Request throughput (req/s): (your number)
Output token throughput (tok/s): (your number)
Median TTFT (ms): (your number)
Median TPOT (ms): (your number)Run the benchmark twice and keep the second result, because the first run pays for warm-up. Then stop the server with Ctrl+C. You now own a baseline file, bf16.json, and a hypothesis: if the cache is the bottleneck, the FP8 run should show a lower median TTFT, since fewer requests sit in the queue.
Steps 9 and 10: Switch to FP8 KV Cache and Check Quality
Step 9: Relaunch with an FP8 cache. One flag changes. Everything else stays identical, which is what makes the comparison fair. The engine documentation lists several storage formats, including fp8, fp8_e4m3, fp8_e5m2, per-token and per-head FP8 variants, plus INT4, INT8 and NVFP4 on some backends. Start with plain fp8. The vLLM quantization API docs describe the weight-side formats that sit next to it.
vllm serve Qwen/Qwen3-8B \
--max-model-len 32768 \
--gpu-memory-utilization 0.92 \
--kv-cache-dtype fp8 \
--port 8000Check the log lines from step 7 again. The cache size should grow substantially, and the maximum concurrency reported at 32,768 tokens per request should rise with it. If the number did not move, the flag was ignored or the attention backend fell back to BF16. The troubleshooting table covers that case. Do not confuse this flag with weight quantization. Shrinking weights, as covered in the GGUF quantization guide, frees memory that the cache can then use, but it does nothing to the per-token cost of the cache itself.
Now rerun the same benchmark from step 8 with a new result name:
vllm bench serve \
--model Qwen/Qwen3-8B \
--dataset-name random \
--random-input-len 4096 \
--random-output-len 256 \
--num-prompts 96 \
--max-concurrency 12 \
--save-result \
--result-filename fp8.jsonCompare the two result files with a tiny script. It prints each metric side by side with the percentage change.
import json
import sys
KEYS = ("request_throughput", "output_throughput", "median_ttft_ms", "median_tpot_ms", "p99_ttft_ms")
with open(sys.argv[1]) as handle:
base = json.load(handle)
with open(sys.argv[2]) as handle:
test = json.load(handle)
print(f"{'metric':22}{sys.argv[1]:>14}{sys.argv[2]:>14}{'change':>10}")
for key in KEYS:
a = base.get(key)
b = test.get(key)
if a is None or b is None or a == 0:
continue
print(f"{key:22}{a:14.2f}{b:14.2f}{(b - a) / a:10.1%}")Run python compare_bench.py bf16.json fp8.json. Read the result honestly. If TTFT dropped and output throughput rose, the cache was your bottleneck and FP8 relieved it. If nothing moved, your load never filled the BF16 pool, or the GPU was compute-bound. Both outcomes are useful, because they tell you whether more cache capacity would help your real traffic.
Step 10: Check that answers did not degrade. Capacity means nothing if quality drops. FP8 storage rounds every key and value, and the error compounds over long contexts. Build a small regression set: create prompts.txt with 40 to 50 varied prompts, one per line, mixing factual questions, code requests and a few long-document tasks. Then generate answers at temperature 0 from the FP8 server.
import json
import sys
from openai import OpenAI
label = sys.argv[1]
client = OpenAI(base_url="http://localhost:8000/v1", api_key="none")
with open("prompts.txt") as handle:
prompts = [line.strip() for line in handle if line.strip()]
results = []
for prompt in prompts:
reply = client.chat.completions.create(
model="Qwen/Qwen3-8B",
messages=[{"role": "user", "content": prompt}],
temperature=0,
max_tokens=200,
)
results.append(reply.choices[0].message.content)
with open(f"outputs_{label}.json", "w") as handle:
json.dump(results, handle)
print(f"saved {len(results)} answers to outputs_{label}.json")Run python gen_outputs.py fp8 now. Then restart the server without the FP8 flag, run python gen_outputs.py bf16a, restart the BF16 server once more, and run python gen_outputs.py bf16b. The third run matters. Even at temperature 0, batching and kernel choices can change the last digits of a calculation, so two BF16 runs rarely match perfectly. That difference is your noise floor, and FP8 only counts as worse if it falls below it.
import json
import sys
with open(sys.argv[1]) as handle:
first = json.load(handle)
with open(sys.argv[2]) as handle:
second = json.load(handle)
identical = 0
prefix_ratios = []
for a, b in zip(first, second):
if a == b:
identical += 1
limit = min(len(a), len(b))
common = 0
while common < limit and a[common] == b[common]:
common += 1
prefix_ratios.append(common / max(len(a), len(b), 1))
total = len(first)
print(f"identical answers: {identical} of {total}")
print(f"average shared prefix: {sum(prefix_ratios) / total:.1%}")python compare_outputs.py outputs_bf16a.json outputs_bf16b.json
python compare_outputs.py outputs_bf16a.json outputs_fp8.jsonIf the FP8 comparison shows a shared-prefix percentage close to the BF16-versus-BF16 one, the format is safe for that model on your prompts. If it falls well below, read a few of the diverging answers by hand. Short drifts in phrasing are harmless. Wrong facts, broken code or lost instructions are not. For a stricter check, run your own task benchmark against both servers, since your prompts matter more than any public leaderboard.
Step 11: Turn On Prefix Caching and Measure Time to First Token
Shrinking the cache is one lever. Reusing it is another. Prefix caching keeps the KV blocks of a prompt prefix, so a later request that begins with the same tokens skips the prefill work for that part. Chatbots with a long system prompt, retrieval apps that paste the same documents, and coding agents that resend a growing transcript all benefit. The same idea powers hosted prompt caching, and the LLM prompt caching guide covers the API-side billing version of it.
Recent vLLM releases turn prefix caching on by default for standard attention models. Passing the flag explicitly does no harm, and it documents your intent in the launch command. The current docs also describe prefix caching for hybrid Mamba checkpoints, where the engine stores prefill state internally to cut time to first token. Restart the server with FP8 and the explicit flag:
vllm serve Qwen/Qwen3-8B \
--max-model-len 32768 \
--gpu-memory-utilization 0.92 \
--kv-cache-dtype fp8 \
--enable-prefix-caching \
--port 8000Now test it. The script below sends three requests that share a long system prompt but end in different questions, and times the first streamed token of each. The first request pays for the full prefill. If caching works, the second and third should answer noticeably faster.
import sys
import time
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="none")
policy = (
"Refunds are issued within 14 days. Shipping takes 3 to 5 business days. "
"Warranty claims need a receipt and a serial number. "
) * 120
def first_token_seconds(question):
start = time.perf_counter()
stream = client.chat.completions.create(
model="Qwen/Qwen3-8B",
messages=[
{"role": "system", "content": policy},
{"role": "user", "content": question},
],
max_tokens=16,
temperature=0,
stream=True,
)
for chunk in stream:
if chunk.choices and chunk.choices[0].delta.content:
return time.perf_counter() - start
return time.perf_counter() - start
for index, question in enumerate(sys.argv[1:] or ["How long do refunds take?", "What does a warranty claim need?", "How long is shipping?"]):
print(f"request {index + 1}: {first_token_seconds(question) * 1000:.0f} ms to first token")python ttft.pyExpect the first line to be the slowest, with the next two clearly lower. The exact milliseconds depend on your GPU, so record your own. If all three times look the same, caching is not hitting. The most common cause is a prefix that changes at the front, such as a timestamp or a per-user ID placed before the shared text. Prefix matching works from the first token forward and stops at the first difference. Put stable content first and variable content last.
Prefix caching also changes how you read capacity. Cached blocks occupy pool space, and the engine evicts the least recently used ones when new requests need room. A busy server with many different long prefixes can churn the cache and see a low hit rate, which the metrics in step 13 expose. Treat hit rate as a product metric, not a bonus.
Steps 12 and 13: Right-Size Context and Monitor the Cache
Step 12: Match limits to real traffic. The default context limit is whatever the model supports, and a high limit does not reserve memory by itself. It does cause startup to fail if the pool cannot hold even one maximum-length request. Measure the length distribution of your actual prompts, then set limits near the 99th percentile instead of the model’s theoretical maximum. The table shows which flag controls what.
| Flag | What it controls | Turn it down when | Turn it up when |
|---|---|---|---|
--max-model-len | Longest prompt plus output per request | Startup says one request cannot fit | Real prompts get rejected as too long |
--gpu-memory-utilization | Share of GPU memory vLLM may claim (default 0.92) | Other processes share the GPU | The GPU is dedicated and the pool is tight |
--max-num-seqs | Maximum sequences scheduled together | Latency spikes under bursts | The pool has spare room and users queue |
--max-num-batched-tokens | Tokens processed per scheduler step | Prefill stalls ongoing decodes | Long prompts arrive in bulk |
--kv-cache-dtype | Storage format of cached keys and values | Quality checks fail | The cache is the bottleneck |
A practical recipe: take the pool size in tokens from the log, divide by the average tokens per request (prompt plus output), and compare the result with your peak concurrency. If the pool covers peak load with 20 percent to spare, you are done. If not, you have four moves in order of cost: enable FP8, cut the context limit, shorten prompts upstream, or add GPU memory. The first three cost nothing.
Step 13: Watch the cache in production. The server exposes Prometheus metrics at /metrics. Metric names differ between releases (cache usage appears as gpu_cache_usage_perc in older builds and kv_cache_usage_perc in newer ones), so filter by the word cache instead of hard-coding a name.
curl -s http://localhost:8000/metrics | grep -i cache
curl -s http://localhost:8000/metrics | grep -i -E "preempt|waiting|running"Alert on three signals. Cache usage above 90 percent for several minutes means you are close to queueing. A rising preemption counter means the engine is evicting running requests and recomputing them, which doubles the cost of those requests. A growing waiting queue with idle compute means the pool, not the GPU cores, limits you. Plot all three next to the prefix cache hit rate and you can explain most latency complaints without guessing.
What October 2026 Open Models Mean for Your Cache
The newest open models push in two directions at once. Context windows keep growing, and architectures keep finding ways to store less per token. DeepSeek shipped V4.1-Flash-1017 as an update to V4.1-Flash, and per NYU Shanghai’s write-up the API price stayed the same. Third-party listings give it a 1M-token window. The official Hugging Face checkpoint is about 510 GB, so this is a multi-GPU model, not a desktop one. DeepSeek’s own pitch is that the architecture trims cache demand, which is why the model can offer a window that the naive formula above would make unaffordable.
The serving numbers show what the data center end looks like. In an October 23 post, the vLLM team reported serving DeepSeek-V4.1-Flash on a GB300 NVL72 rack at 954K tokens per second per GPU, with a P90 of 196 tokens per second per user, which they put at 7.9 times their earlier single-node result. They used prefill/decode disaggregation and wide expert parallelism. A second vLLM post covered GLM-5.3 on 16 H200 GPUs, reaching 1M-token contexts. None of that fits on one card, and the point is the shape of the solution: split the work across GPUs so the cache has somewhere to live.
At the other end, NVIDIA released Nemotron 3.5 Bolt 74B-A9B on October 15, built to run locally on RTX and DGX Spark systems. NVIDIA describes a native 2-bit quantization that fits the model in 24 GB, and says vLLM supports it. That solves the weights problem, but not the cache problem. Whatever memory remains after the weights still has to hold every active session. The same company released Nemotron 3.5 Super and Lightning on October 14 as open hybrid Mamba-Transformer models. Hybrid designs hold a cache only in their attention layers, so their per-token cost is lower than a pure transformer of the same size. Run kvbudget.py (in the project section) and it warns you when a config looks hybrid.
For application builders, the practical effect is a shift in what to budget. A year ago the question was whether the weights fit. Now the question is how many concurrent long sessions fit beside them. If you build on hosted 1M-context endpoints instead, the guide on the Qwen3.8 Flash API shows how those limits surface in billing. For wider context on this beat, the AI and machine learning section tracks new releases as they land.
| Model or system | Date | Cache-relevant fact |
|---|---|---|
| DeepSeek-V4.1-Flash-1017 | Oct. 17, 2026 | Update to V4.1-Flash (1M window per model listings), ~510 GB official checkpoint |
| DeepSeek V4.1 architecture | Sept. 10, 2026 | About 1/4 the HBM and 1/8 the SSD for cache vs. prior generation (DeepSeek) |
| Nemotron 3.5 Bolt 74B-A9B | Oct. 15, 2026 | Native 2-bit weights fit 24 GB, cache still grows per session |
| Nemotron 3.5 Super / Lightning | Oct. 14, 2026 | Hybrid Mamba-Transformer, cache only in attention layers |
| vLLM 0.32.0 | By Oct. 19, 2026 | Release headline: low-precision KV cache everywhere |
The Ollama Path: KV Cache Quantization on One Machine
Not everyone needs a serving engine. If you run a model for yourself, Ollama gives you the same two levers through environment variables. Flash attention must be on for a quantized cache to work. Then you pick a cache type: f16 is the default, q8_0 stores roughly half the bytes, and q4_0 stores roughly a quarter. The step-by-step install is in the guide to running Ollama locally, so only the cache settings appear here.
export OLLAMA_FLASH_ATTENTION=1
export OLLAMA_KV_CACHE_TYPE=q8_0
export OLLAMA_CONTEXT_LENGTH=32768
ollama serveIn a second terminal, pull a model and check what the runtime allocated. Confirm the exact tag in the Ollama model library first, since tags change. Then run the model and ask ollama ps how much memory it holds.
ollama pull qwen3:8b
ollama run qwen3:8b "Summarize why the KV cache grows with context."
ollama psRun ollama ps once with f16 and once with q8_0, at the same context length. The reported size should shrink by about the cache’s saved share. Ollama’s OpenAI-compatible endpoint covers a subset of the OpenAI API, so the Python scripts from steps 10 and 11 work with a changed base_url of http://localhost:11434/v1, though you should expect different timing. Prefer q8_0 over q4_0 unless memory is desperate. Four-bit caches degrade long-context recall sooner, and you only notice after a user complains.
The same idea exists outside Ollama. The llama.cpp server accepts cache-type flags for keys and values, and the llama.cpp tutorial shows where to place them. As a rule, vLLM suits shared servers where many users hit one model, and Ollama or llama.cpp suit one person on one machine. Cache quantization helps both, but only vLLM exposes the pool and the hit rate for production monitoring.
Complete Working Project: A KV Budget Planner and A/B Harness
Time to combine the pieces. The project below is one planning script plus the A/B routine from steps 8 to 10. You give the planner a model, your GPU size and your weight footprint. It reads the config, prints a cache budget for BF16 and FP8, flags architectures that break the simple formula, and states a verdict on what to launch. Then the harness verifies the prediction on real hardware. Create this folder layout:
kvlab/
kvbudget.py planner (below)
prompts.txt 40 to 50 regression prompts
gen_outputs.py step 10
compare_outputs.py step 10
compare_bench.py step 9
ttft.py step 11#!/usr/bin/env python3
import argparse
import json
import os
GIB = 1024 ** 3
DTYPES = {"bf16": 2, "fp8": 1}
CONTEXTS = (4096, 8192, 32768, 131072, 262144, 1048576)
def load_config(source):
if os.path.isdir(source):
path = os.path.join(source, "config.json")
else:
from huggingface_hub import hf_hub_download
path = hf_hub_download(source, "config.json")
with open(path) as handle:
raw = json.load(handle)
return raw.get("text_config", raw)
def per_token_bytes(cfg, element_bytes):
heads = cfg["num_attention_heads"]
kv_heads = cfg.get("num_key_value_heads", heads)
head_dim = cfg.get("head_dim") or cfg["hidden_size"] // heads
return 2 * cfg["num_hidden_layers"] * kv_heads * head_dim * element_bytes
def warn_if_unusual(cfg):
if "kv_lora_rank" in cfg:
print("WARNING: latent attention found, the real cache is smaller than this estimate.")
if cfg.get("sliding_window"):
print("WARNING: sliding window found, long contexts may need less cache.")
if "layer_types" in cfg or "layers_block_type" in cfg:
print("WARNING: mixed layer types, only attention layers hold a KV cache.")
def main():
parser = argparse.ArgumentParser(description="Estimate vLLM KV cache capacity")
parser.add_argument("model", help="Hugging Face repo id or local folder with config.json")
parser.add_argument("--gpu-gib", type=float, default=24.0)
parser.add_argument("--util", type=float, default=0.92)
parser.add_argument("--weights-gib", type=float, required=True)
parser.add_argument("--overhead-gib", type=float, default=2.0)
parser.add_argument("--target-context", type=int, default=32768)
args = parser.parse_args()
cfg = load_config(args.model)
warn_if_unusual(cfg)
pool = args.gpu_gib * args.util - args.weights_gib - args.overhead_gib
if pool <= 0:
raise SystemExit("No room for a cache: raise --gpu-gib or shrink the weights.")
print(f"cache pool: {pool:.2f} GiB")
print()
print(f"{'dtype':6}{'KiB/token':>11}{'pool tokens':>13}{'slots at ' + str(args.target_context):>20}")
best = None
for name, size in DTYPES.items():
per_token = per_token_bytes(cfg, size)
tokens = int(pool * GIB / per_token)
slots = tokens / args.target_context
print(f"{name:6}{per_token / 1024:11.1f}{tokens:13,}{slots:20.2f}")
if slots >= 1 and best is None:
best = name
print()
print("per-request cache by context length")
for ctx in CONTEXTS:
cells = [f"{ctx:>10,} tokens"]
for name, size in DTYPES.items():
cells.append(f"{name} {ctx * per_token_bytes(cfg, size) / GIB:8.2f} GiB")
print(" ".join(cells))
print()
if best is None:
print("Verdict: not even one request fits. Shorten --max-model-len or use a larger GPU.")
elif best == "bf16":
print("Verdict: BF16 cache holds the target context. FP8 buys you more concurrency.")
else:
print("Verdict: only the FP8 cache holds the target context. Add --kv-cache-dtype fp8.")
if __name__ == "__main__":
main()Run it for the example config on a 24 GiB card, with the weight footprint from step 5:
python kvbudget.py Qwen/Qwen3-8B --gpu-gib 24 --weights-gib 15.3 --target-context 32768cache pool: 4.78 GiB
dtype KiB/token pool tokens slots at 32768
bf16 144.0 34,806 1.06
fp8 72.0 69,613 2.12
per-request cache by context length
4,096 tokens bf16 0.56 GiB fp8 0.28 GiB
8,192 tokens bf16 1.12 GiB fp8 0.56 GiB
32,768 tokens bf16 4.50 GiB fp8 2.25 GiB
131,072 tokens bf16 18.00 GiB fp8 9.00 GiB
262,144 tokens bf16 36.00 GiB fp8 18.00 GiB
1,048,576 tokens bf16 144.00 GiB fp8 72.00 GiB
Verdict: BF16 cache holds the target context. FP8 buys you more concurrency.This output comes from the example 36-layer, 8-KV-head, 128-dimension config. Your model will print different rows. The workflow around it is the real deliverable: run the planner, launch the server it recommends, compare the log line against the planner’s pool figure, then run the A/B steps from this guide whenever you change the flags, the model or the vLLM version. Commit the result files next to your deployment config, and the next engineer inherits your measurements instead of your opinions.
One last habit makes the project stick. When a new model lands, run kvbudget.py on its config before you download 100 GB of weights. A thirty-second check tells you whether your hardware can serve the context window on the model card or only a fraction of it.
Common Pitfalls That Waste an Afternoon
- Trusting the model card’s context length. A 1M-token window describes what the architecture can address, not what your GPU can hold. Compute the cache for your target length before you promise anyone that length.
- Raising utilization to fix everything. Pushing
--gpu-memory-utilizationtoward 1.0 can crash the first time activation memory spikes. Leave headroom unless the GPU is dedicated and you have load-tested the setting. - Comparing runs on a dirty GPU. A stray process holding 2 GiB changes the pool size and invalidates the comparison. Check
nvidia-smibefore every launch. - Skipping the noise floor. Two BF16 runs at temperature 0 can differ. Without that baseline you will blame FP8 for drift it did not cause.
- Putting variable text at the front of the prompt. A timestamp or user ID before the shared system prompt kills the prefix cache hit rate. Order prompts stable first, variable last.
- Assuming FP8 works for every model and backend. NVIDIA documents that Nemotron Super V3 with FP8 needs the Triton attention backend, so support depends on the model and the kernel path. Test each model.
- Benchmarking with one request at a time. A single request never fills the pool, so every cache setting looks identical. Use concurrency that exceeds the BF16 pool, as in step 8.
- Forgetting that the formula changes for latent attention. For models with compressed caches, the simple estimate overshoots. Treat it as an upper bound and trust the log line.
Troubleshooting: 10 Errors and Fixes
These are the failures you are most likely to hit while following this guide. Start with the symptom, then apply the matching fix. The NVIDIA NIM release notes document a related real-world case: in one BF16 multimodal profile, a decoder memory reservation of 2.05 GiB exceeded the available KV cache budget and stopped the server from starting. The lesson holds everywhere: when startup fails on memory, read the cache budget first.
| # | Symptom | Likely cause | Fix |
|---|---|---|---|
| 1 | Startup error: max sequence length larger than what the KV cache can store | Pool smaller than one maximum-length request | Lower --max-model-len, add --kv-cache-dtype fp8, or raise utilization |
| 2 | CUDA out of memory during weight load | Weights plus other GPU processes exceed free memory | Close other GPU apps, use a smaller or quantized model |
| 3 | CUDA out of memory right after the cache is sized | Utilization too high for activation spikes | Drop --gpu-memory-utilization by 0.02 to 0.05 and retry |
| 4 | FP8 flag accepted but cache size did not double | Attention backend fell back to BF16 | Check logs for backend messages, try another attention backend, confirm model support |
| 5 | Error naming an unsupported KV dtype | GPU or backend lacks that format | Try fp8_e4m3 or fp8_e5m2, or stay on BF16 |
| 6 | Answers worse after FP8 | Rounding error on a sensitive model or long inputs | Re-run step 10, test the other FP8 variant, or revert for that model |
| 7 | Prefix cache hit rate near zero | Shared text is not at the start of the prompt | Move variable content to the end, remove timestamps from system prompts |
| 8 | Latency spikes under load, preemption counter rising | Pool exhausted, engine evicts and recomputes | Lower --max-num-seqs, enable FP8, shorten prompts, or add GPU memory |
| 9 | First launch hangs for minutes | Kernels compiling at startup | Run vllm download-kernels after installing |
| 10 | Port 8000 already in use | Old server still running | Stop it, or pass --port 8001 and update client URLs |
Two more checks save time. If the model download stalls or returns an authorization error, confirm that HF_TOKEN is exported in the same shell that runs vllm serve and that you accepted the model’s license on its page. If metrics output shows no cache gauge, your build uses a different name, so grep for the word cache without a prefix as shown in step 13.
Advanced Tips for Long-Context Serving
Share the GPU with training jobs carefully
Reinforcement learning loops often run vLLM for rollouts next to a trainer on the same card. Each process claims its fraction of memory, and utilization is per instance, so two engines at 0.5 each leave little room for the trainer. The GRPO tutorial on one GPU shows a split that works, and the planner here helps you pick the rollout engine’s share.
Scale out when one GPU cannot hold the cache
Tensor parallelism splits weights and cache across GPUs, which raises the total pool at the cost of interconnect traffic. Current vLLM documentation also describes hybrid data-parallel load balancing, where an API server balances across local ranks while an external balancer spreads work across nodes. For the largest models, the vLLM team’s DeepSeek and GLM posts show prefill/decode disaggregation, where one set of GPUs handles prompts and another handles generation. Reach for these only after FP8 and context right-sizing stop helping.
Lock in a regression suite before upgrades
Engine releases change defaults, kernels and metric names. Keep the benchmark files and the prompt set in version control, and rerun the comparison whenever you bump vLLM or swap a model. A thirty-minute check beats a production surprise. Pin the version in your container image, record the launch flags in the same repo, and note the pool size from the log next to each deploy.
FAQ: KV Cache and vLLM
What is a KV cache in an LLM?
It is the stored set of key and value vectors for every token the model has already processed. Attention reads it on each new token instead of recomputing it. The cache speeds up generation but takes GPU memory in proportion to context length and the number of concurrent requests.
How do I calculate KV cache size?
Multiply 2 by the layer count, the KV head count, the head dimension and the bytes per element to get bytes per token. Then multiply by tokens per request and by concurrent requests. The formula fits standard attention models only. Latent-attention and hybrid models store less than it predicts.
Does FP8 KV cache hurt accuracy?
It can, depending on the model, the backend and the prompt length. Rounding error is small per token but can add up across long contexts. Run the step 10 comparison against your own prompts, and measure the BF16-versus-BF16 noise floor first so you can separate real loss from ordinary variation.
Is prefix caching on by default in vLLM?
Recent releases turn it on for standard attention models, and the docs describe separate handling for hybrid Mamba checkpoints. Check the startup log of your build and pass --enable-prefix-caching to make the intent explicit. Verify the effect with the TTFT script in step 11.
Can I run a 1M-token model on one consumer GPU?
Usually not at the full window. The official DeepSeek-V4.1-Flash-1017 checkpoint is about 510 GB, and even with compressed caches a million tokens of history is a data-center workload. You can serve smaller models with shorter limits, or use hosted endpoints for the full window.
What is the difference between weight quantization and KV cache quantization?
Weight quantization shrinks the model file and the memory it occupies. KV cache quantization shrinks the per-token memory used while generating. They stack. Smaller weights leave a bigger pool, and a quantized cache stretches that pool further.
Which vLLM version should I use for this guide?
This guide pins vLLM 0.32.0, the line published by October 19, 2026. Older lines such as 0.31.0 still work for most steps, but flag names and metric names may differ. Pin whichever version you choose and record it with your results.
