✨ อัปเดตล่าสุด: tensorrt llm vs vllm

With default settings, vLLM beat TensorRT-LLM by 57% at 32 concurrent requests. One line in the TensorRT-LLM config flipped that: switching its KV cache to FP8 put it 14% ahead of the best vLLM run. It also stayed 7% faster for a single user. Both servers loaded the same NVFP4 checkpoint of Qwen3.8-27B on the same RTX 5090, and the same streaming client hit both.

Original content from computingforgeeks.com - post 173806

Most TensorRT-LLM vs vLLM benchmarks run on H100s and quote vendor numbers. This one is on a 32 GB consumer card, where memory decides the result. It covers tokens per second at 1 to 32 concurrent requests, time to first token, KV cache room, cold-start time and GSM8K parity. It also shows the log lines that explain the worst run of each engine, and the error you get trying to run TensorRT-LLM text-only on this model. The runs used TensorRT-LLM 1.3.0rc29 and vLLM 0.31.0 on an RTX 5090 with the 595 driver in October 2026.

TensorRT-LLM vs vLLM results on the RTX 5090

Each throughput cell is aggregate output tokens per second across all concurrent requests, from the first measured round at that concurrency. Every request sent a 474-token prompt and generated exactly 256 tokens.

Engine and KV cache	1	4	8	16	32	TTFT at 1 (median)	KV room (as logged)	Start time
TensorRT-LLM, KV default (BF16)	69.7	230	485	852	759	121 ms	17,088 tokens	172 s
TensorRT-LLM, KV FP8	70.6	228	483	854	1,361	123 ms	34,144 tokens	142 s
vLLM, KV default (BF16)	65.6	219	440	778	1,195	177 ms	11.3 full-length requests	718 s (first start)
vLLM, KV FP8	63.2	220	435	753	776	169 ms	14.2 full-length requests	202 s

The KV column uses the unit each engine logs. TensorRT-LLM reports attention KV tokens. vLLM reports how many requests of the full 4,096 tokens its block pool can hold (its log prints that number times 4,096 as “GPU KV cache size”). The two are explained in the sections below.

Per-request decode speed tells the single-user story more cleanly. TensorRT-LLM held 73.0 tokens per second on one request against 68.5 for vLLM. Both lost about a third of that by 32 requests. Repeat rounds agreed within about 1%, with two exceptions. The first TensorRT-LLM round at 4 requests ran 10% below the later ones (a warm-up effect). The collapsed default-KV run at 32 ranged from 759 to 818. A second RTX 5090 in another data center, rented later the same day, ran the TensorRT-LLM FP8 KV config at 1,409 tokens per second at 32 requests, 3.5% above this box. Treat gaps under about 4 to 6% as host noise.

How the two engines were tested

The model is nvidia/Qwen3.8-27B-NVFP4, a 21.9 GB ModelOpt checkpoint with NVFP4 MLP weights and FP8 attention. Qwen3.8 is a hybrid model that mixes Gated DeltaNet linear-attention layers with regular attention, and that matters for memory. A 27B model only fits a 32 GB card in either engine with 4-bit weights. This is the one 4-bit format both engines load. The TensorRT-LLM support matrix lists no W4A16 INT4 kernels for SM120. That rules out the compressed-tensors INT4 checkpoint we used for vLLM in the Ollama, vLLM and llama.cpp comparison.

Both servers got 32 sequence slots of 4,096 tokens, the same limits as that comparison. The client is also the same one: a Python script that streams /v1/completions with temperature 0 and ignore_eos. It sends one discarded warm-up request, then runs three rounds at 1 and 4 concurrent requests and two rounds at 8, 16 and 32. TTFT is the time to the first streamed token.

Everything ran inside the TensorRT-LLM container, set up as in the TensorRT-LLM install guide with the host directory ~/trtllm mounted at /workspace. Download the checkpoint from a shell in the container. The image already ships the hf CLI. /root/models lives in the container filesystem and goes away with the container, so use /workspace/models if you want to keep the 22 GB download:

hf download nvidia/Qwen3.8-27B-NVFP4 --local-dir /root/models/Qwen3.8-27B-NVFP4

Server settings go in a YAML file. Create it on the host so the container sees it under /workspace:

vim ~/trtllm/q38-fp8kv.yaml

This is the exact file for the winning run. The default-KV run used the same file without the dtype: fp8 line:

max_batch_size: 32
max_num_tokens: 8192
max_seq_len: 4096
trust_remote_code: true
kv_cache_config:
  free_gpu_memory_fraction: 0.85
  enable_block_reuse: false
  dtype: fp8
cuda_graph_config:
  enable_padding: true
  max_batch_size: 32

Block reuse is off because the TensorRT-LLM supported-models table lists KV cache reuse as unsupported for this architecture. Start the server:

trtllm-serve /root/models/Qwen3.8-27B-NVFP4 --host 0.0.0.0 --port 8000 --config /workspace/q38-fp8kv.yaml

vLLM went into its own virtual environment inside the same container, so both engines read the same files and the same port mapping applies. Install the release used in the earlier comparison:

pip install -q uv && uv venv -q -p 3.12 /root/vllm-env
uv pip install -q -p /root/vllm-env/bin/python vllm==0.31.0 --torch-backend=auto #https://github.com/vllm-project/vllm/releases

Stop trtllm-serve with Ctrl+C first, because both servers want port 8000 and most of the GPU. Then start vLLM. --language-model-only skips the vision encoder this checkpoint carries, and --kv-cache-dtype fp8 was added only for the FP8 row:

source /root/vllm-env/bin/activate
vllm serve /root/models/Qwen3.8-27B-NVFP4 --served-model-name m --language-model-only \
  --max-model-len 4096 --max-num-seqs 32 --gpu-memory-utilization 0.92 --port 8000

The vLLM install guide covers running vLLM on its own host instead.

Why TensorRT-LLM collapsed at 32 requests with default KV

It ran out of KV cache. Aggregate throughput dropped from 852 tokens per second at 16 requests to 759 at 32, and each round took 10 to 11 seconds instead of about 6. The server log shows how little room was left:

[10/12/2026-09:48:53] [TRT-LLM] [I] [_torch] Memory used at after loading weights (inside torch) in memory usage profiling: 21.54 GiB
[10/12/2026-09:49:23] [TRT-LLM] [I] [_torch] Peak memory during memory usage profiling (torch + non-torch): 30.13 GiB, available KV cache memory when calculating max tokens: 3.45 GiB, fraction is set 0.85, kv size per token is 65536 bytes/token + 2588737536 bytes fixed cost. device total memory 31.36 GiB, temporary kv cache memory during profiling 2.84 GiB
[TRT-LLM] [I] [batchmgr][RANK 0] [MemUsageChange] Allocated 3.45 GiB for max tokens in paged KV cache (18144).

The weights took 21.54 GiB, against 19.05 GiB for the same checkpoint in vLLM text-only mode. The tensor index of the checkpoint explains about two-thirds of that gap: the vision encoder is 0.86 GiB and the multi-token-prediction head is 0.79 GiB. TensorRT-LLM most likely loaded both. That left 3.45 GiB for the cache, and 2.4 GiB of it is the fixed cost in the log: Gated DeltaNet state slots for the running sequences, which a pure attention model would not need. About 1.0 GiB remained for attention KV.

The 18,144 in the last line counts the state slots too. The allocation behind it is 534 attention blocks of 32 tokens, which is 17,088 tokens. A request of 474 prompt and 256 output tokens needs 23 blocks, so 23 requests fit and 9 of the 32 waited. The default scheduler policy, GUARANTEED_NO_EVICT, only admits requests it can finish without evicting anything. FP8 KV halves the per-token cost to 32 KiB, which gave 1,067 blocks (34,144 tokens). All 32 requests then ran in one pass, and throughput jumped to 1,361 tokens per second.

Error: ‘NoneType’ object has no attribute ‘load_weights’ with disable_mm_encoder

The fair fix would be to run TensorRT-LLM text-only, like vLLM. The current release has a prototype option for that, disable_mm_encoder, and its docstring names the Qwen3.5-VL family that Qwen3.8 belongs to. We rented a second RTX 5090 and added disable_mm_encoder: true to both configs. The server accepted the key, then failed while loading weights:

[10/12/2026-11:34:01] [TRT-LLM] [I] [llmapi] Setting multimodal_config.encoder_cache_max_bytes to 0 because disable_mm_encoder=True.
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/_torch/models/modeling_qwen_image_bench.py", line 82, in load_weights
    self.mm_encoder.load_weights(weights)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'load_weights'

The option removes the encoder object, but the Qwen3.5 weight loader still calls into it. That’s a bug in this release, not a config mistake. It failed the same way with and without FP8 KV. Remove the line to get a working server. Until a release fixes it, TensorRT-LLM carries about 2.5 GiB more weights than vLLM on this checkpoint, 1.65 GiB of it the vision encoder and MTP head. So in this test it won at 32 requests despite running with less memory, not because of a more generous setup.

Why vLLM got slower with FP8 KV cache

The same switch made vLLM worse. At 32 requests it fell from 1,195 to 776 tokens per second, on both rounds. The cause is how vLLM pages a hybrid model. Attention KV and the Gated DeltaNet state share one block pool, and every page must be the same size in bytes as a state page. Halving the bytes per token therefore doubles the tokens per page instead of saving memory:

(EngineCore pid=52138) INFO 10-12 09:52:25 [interface.py:989] Setting attention block size to 784 tokens to ensure that attention page size is >= mamba page size.
(EngineCore pid=158254) INFO 10-12 10:10:10 [interface.py:989] Setting attention block size to 1568 tokens to ensure that attention page size is >= mamba page size.

The first line is the BF16 KV run, the second the FP8 one. Each running request holds four pages either way (one per KV cache group), so a short request gets no cheaper. Meanwhile the FP8 run switched attention backend from FLASH_ATTN to FLASHINFER, and memory available for the pool fell from 6.53 to 6.14 GiB. Working back from the logged concurrency figures, the pool shrank from about 136 pages to about 128. That is room for 33 requests with BF16 KV and 31 with FP8. The engine stats from the FP8 32-request rounds match:

Running: 31 reqs, Waiting: 1 reqs, GPU KV cache usage: 97.6%

One request was left waiting, so every 32-request round turned into two back-to-back batches. With BF16 KV, the log shows all 32 running at 94.8% usage. The backend switch is also a likely reason single-request speed dropped from 68.5 to 65.7 tokens per second. For a hybrid model on this card, leave vLLM on its default KV dtype. That is the opposite of the TensorRT-LLM advice, which is why one setting can produce either result.

Both engines print the numbers to check at startup:

TensorRT-LLM and vLLM KV cache capacity and attention block size log lines on RTX 5090

Cold start: 172 seconds against 718

TensorRT-LLM reached a healthy endpoint 172 seconds after its first launch with this model. It had served a smaller model on the same box earlier, so some of its own caches were already warm. vLLM had never run there, and its first start took 718 seconds. Most of that went to FlashInfer compiling Blackwell kernels:

2026-10-12 09:53:26,059 - INFO - core.py:564 - flashinfer.jit: Building JIT module gemm_sm120; this can take several minutes on first use.
2026-10-12 09:57:11,232 - INFO - core.py:564 - flashinfer.jit: Building JIT module fp4_gemm_cutlass_sm120; this can take several minutes on first use.
(EngineCore pid=52138) INFO 10-12 10:01:13 [model_runner.py:1087] Graph capturing finished in 136 secs, took 0.18 GiB
(EngineCore pid=52138) INFO 10-12 10:03:20 [core.py:373] init engine (profile, create kv cache, warmup model) took 655.77 s (compilation: 59.47 s)

The FP8 KV start took 202 seconds because it needed new FP8 attention kernels and a fresh compile. A real warm restart of the original vLLM config was much faster. Its log runs 75 seconds from the first line to “Starting vLLM server”, with engine init at 46.68 seconds. Adding the roughly 16 seconds before the first log line gives about 90 seconds from launch, which beats the 142-second TensorRT-LLM warm start. The compiled kernels are cached on disk, so the slow start is a once-per-node cost. For autoscaling, keep the vLLM cache on a persistent volume, or budget twelve minutes for every fresh node.

Accuracy is identical on the same weights

Both engines serve the same NVFP4 tensors, so GSM8K should come out the same, and it did. We ran lm-evaluation-harness against the completions endpoint of each server, 5-shot, all 1,319 questions:

Server and KV cache	Strict-match	Flexible-extract
TensorRT-LLM, FP8 KV	54.28%	58.98%
vLLM, FP8 KV	54.36%	59.74%
vLLM, BF16 KV	56.25%	59.51%

Each score carries a standard error of about 1.36 points, so the largest gap (1.97 points on strict-match) is about one combined standard error (1.9 points) and not significant. The two engines agree almost exactly at the same KV setting. FP8 KV showed no significant loss in vLLM. We did not run a BF16 KV GSM8K on TensorRT-LLM, so the same conclusion there rests on the vLLM pair and the matching FP8 scores.

The low absolute score is the harness, not the model. Raw few-shot prompting makes Qwen3.8 open a thinking block, and the default 256-token answer cap cuts many answers off before the final number. The NVIDIA reference for the BF16 model is about 90%, measured with the chat template and thinking disabled. Read this table as engine parity only: neither server corrupts the NVFP4 model. An 8B model fits this card in BF16 and FP8 too, and the NVFP4 vs FP8 vs BF16 test measures what the 4-bit format itself costs.

The two settings that decided this benchmark

Neither engine won on default settings in every column. On a 32 GB card with a hybrid model, the result came down to two values:

TensorRT-LLM kv_cache_config.dtype: fp8: doubles the attention KV room left after the weights and the Gated DeltaNet state, turning the slowest 32-request run into the fastest. At startup, find the blocks_per_window {4096: (N, 0) entry in the log, multiply N by 32, and compare it with your concurrency times prompt plus output. If it is smaller, requests will queue. The Allocated line overstates the room because it also counts the state slots.
vLLM --kv-cache-dtype left on auto: FP8 doubles the tokens per page on hybrid models instead of saving memory, and on this card the pool lost room for two requests. Check the Setting attention block size line, and watch Waiting: in the engine stats during load.

With both set right, TensorRT-LLM was 7% faster for one user and 14% faster at 32, while carrying the vision encoder vLLM skipped. vLLM is still the easier server to run: one pip install, broader model coverage, a text-only mode that works on this checkpoint, and faster warm restarts. Need more than two engines in the picture? The SGLang vs vLLM benchmark and the vLLM vs llm-d comparison cover the other serving options we have tested.