Ollama vs vLLM vs llama.cpp: Tokens Per Second on the Same GPU

Ollama runs one request at a time on the newest Qwen models. That is the headline from re-running our Ollama vs vLLM vs llama.cpp benchmark, and it changes the answer for anyone serving more than one user. Send it 32 requests on an RTX 4090 and the median request waits over a minute and a half for its first token. The last one waits well over three minutes.

Original content from computingforgeeks.com - post 171658

The first version of this test used Qwen2.5-7B. That no longer reflects what people run on a 24 GB card. So we rebuilt the lab around Qwen3.8-27B, a hybrid attention and linear-attention model with a built-in multi-token prediction head. Same three GPUs, same rules: one card per box, one client, one prompt, three engines. The numbers below cover single-stream decode, aggregate throughput from 1 to 32 concurrent requests, speculative decoding in both Ollama and vLLM, and what breaks when a 27B model meets 24 GB. Every figure was measured on rented RTX 4090, L40S and RTX 5090 cards in October 2026 with vLLM 0.31.0, llama.cpp v0.6.0 and Ollama 0.35.1.

Ollama vs vLLM vs llama.cpp results

Single stream means one request in flight, reported as the median decode rate in tokens per second. The 32-concurrent column is aggregate output throughput across all 32 requests. Ollama appears twice: with its speculative draft disabled (a like-for-like comparison) and with the default library settings, which run the MTP draft head built into the model. The draft-off runs pass {"draft_num_predict": 0} in the request options.

GPU	Engine	Single stream (tok/s)	32 concurrent (tok/s)	Time to first token at 32
RTX 4090 24 GB	vLLM	54.3	300	8.6 s
RTX 4090 24 GB	llama.cpp	49.5	306	6.5 s
RTX 4090 24 GB	Ollama, draft off	39.9	39	101.7 s
RTX 4090 24 GB	Ollama, default MTP draft	76.2	not run	not run
L40S 48 GB	vLLM	43.7	554	3.0 s
L40S 48 GB	llama.cpp	39.1	241	6.4 s
L40S 48 GB	Ollama, draft off	20.0	20	200.3 s
L40S 48 GB	Ollama, default MTP draft	57.3	not run	not run
RTX 5090 32 GB	vLLM	85.7	936	2.3 s
RTX 5090 32 GB	llama.cpp	77.8	369	6.6 s
RTX 5090 32 GB	Ollama, draft off	76.8	74	54.1 s
RTX 5090 32 GB	Ollama, default MTP draft	129.7	not run	not run

Three things stand out. Without speculative decoding, vLLM wins single-stream decode on every card by a small margin and wins concurrency by a wide one once it has VRAM to work with. llama.cpp scales well and is the most predictable of the three on a 24 GB card. Ollama does not scale at all on this model, and on the L40S it decodes at half the speed of llama.cpp serving the identical file.

How we tested

Each GPU was a separate box. We rented them on vast.ai, running Ubuntu 24.04 with the 595 NVIDIA driver branch and CUDA 13.2. A fourth RTX 4090 on the older 570 driver ran the Ollama tests and llama-bench only, to check a driver theory from the first version of this article. All three engines ran on the same physical GPU in turn, never at the same time.

The engines got the weights each one is built for. All three are 4-bit:

Engine	Weights	Size on disk
llama.cpp	unsloth Qwen3.8-27B GGUF, UD-Q4_K_M	16.5 GB
Ollama	library qwen3.8:27b (Q4_K_M GGUF plus vision projector)	17 GB
Ollama, second run	the same unsloth GGUF as llama.cpp, imported with a Modelfile	16.5 GB
vLLM	RedHatAI Qwen3.8-27B INT4 (W4A16, GPTQ with AWQ smoothing)	19.5 GB

The prompt is a fixed 474-token incident-runbook request sent through the raw Qwen chat template with thinking disabled. Temperature is 0. Output is capped at 256 tokens. vLLM and llama.cpp get ignore_eos; Ollama has no such option, but the prompt asks for a long answer and every Ollama request still produced the full 256 tokens. One Python client streams from all three: OpenAI-style /v1/completions for vLLM and llama.cpp, and /api/generate in raw mode for Ollama. Time to first token is measured at the client from send to first streamed token. Decode rate is 255 tokens divided by the time from first to last token. Aggregate throughput is total tokens divided by wall time for the whole wave. Each concurrency level ran three waves at 1 and 4 requests and two waves above that, after one discarded warm-up request, and the tables show the median.

Every request carries the same prompt, and the engines do not treat that equally. llama.cpp reuses the cached prefix, so after the first request it evaluates only 4 new prompt tokens. vLLM reports a 0.0 percent prefix cache hit rate in every run: the 474-token prompt never fills one of its 1,568-token hybrid attention blocks, so it prefills the whole prompt each time. That flatters llama.cpp on time to first token. Real traffic with distinct prompts pays full prefill on every engine.

Context is 4,096 tokens per request with room for 32 in flight. vLLM got this launch line:

vllm serve ~/models/int4 --served-model-name m --language-model-only --max-model-len 4096 --max-num-seqs 32 --gpu-memory-utilization 0.92 --port 8000

The --language-model-only flag matters. The INT4 checkpoint ships as Qwen3_5ForConditionalGeneration with the vision tower attached, and skipping it frees VRAM that a 24 GB card cannot spare. llama.cpp ran with 32 slots and flash attention:

llama-server -m ~/models/gguf/Qwen3.8-27B-UD-Q4_K_M.gguf -ngl 999 -fa on -c 131072 -np 32 --host 127.0.0.1 --port 8080

That line runs out of memory on the RTX 4090, so the 4090 used -c 32768 -np 32 instead (1,024 tokens per slot, enough for the 730 tokens each request needs). Ollama got the matching environment:

OLLAMA_NUM_PARALLEL=32 OLLAMA_CONTEXT_LENGTH=4096 OLLAMA_FLASH_ATTENTION=1 ollama serve

Compare single-stream speed

With one request in flight, decode is bound by memory bandwidth. All three engines read roughly the same 4-bit weights per token, so they should land close together. vLLM and llama.cpp do. The W4A16 Marlin kernels in vLLM came out 9 to 12 percent ahead of llama.cpp on all three cards.

Ollama is the outlier, and how far it falls behind depends on the card. On the RTX 5090 it matched llama.cpp within two percent. On the RTX 4090 it decoded 19 percent slower. On the L40S it ran at about half the llama.cpp rate: 20.0 tokens per second against 39.1, serving the same GGUF file. The reference figure from llama-bench on the L40S was 40.1. The driver section below covers what we ruled out.

Push 32 concurrent requests

Concurrency is where the engines split. On the RTX 5090, vLLM climbed from 82 tokens per second at one request to 936 at 32, about eleven times its single-request throughput and two and a half times what llama.cpp reached. Its time to first token at 32 requests was 2.3 seconds, the lowest of any engine on any card.

llama.cpp scaled steadily on every GPU: 306 tokens per second at 32 requests on the RTX 4090, 241 on the L40S, and 369 on the RTX 5090. Per-request speed drops as slots fill (12.6 tokens per second each at 32 on the 4090), but nobody waits long for a first token.

The RTX 4090 tells a different story for vLLM. It reached 302 tokens per second at 8 requests and then stopped. It sat at 300 for 16 and 32 requests, and at 32 llama.cpp edged ahead with 306. The reason is in the startup log:

(EngineCore pid=12528) INFO 10-05 20:51:57 [kv_cache_utils.py:2464] GPU KV cache size: 17,294 tokens, Maximum concurrency for 4,096 tokens per request: 4.22x

The weights load as 16.84 GiB, and after activations and CUDA graphs the 4090 has 1.85 GiB left for cache. Qwen3.8 is a hybrid model, so vLLM pads its attention block size to 1,568 tokens to line pages up with the linear-attention state. Each of our 730-token requests still occupies a full block. During the 32-request wave the scheduler log never showed more than nine requests running, with KV cache usage at 97.3 percent and the rest queued. The L40S, with 48 GB, got a 199,793-token cache, and its log shows all 32 requests running with nothing waiting.

Why Ollama runs one request at a time

Aggregate throughput from Ollama was flat. On the RTX 4090 it delivered 39.1 tokens per second at 1 request and 39.1 at 32. Time to first token grew linearly instead: 10 seconds at 4 requests, 23 at 8, 49 at 16 and 101.7 at 32. That is a queue, not a batch. The server log says so directly, even with OLLAMA_NUM_PARALLEL=32 set:

Ollama classifies Qwen3.8 as the qwen35 architecture. It refuses parallel slots for it, then launches its bundled llama-server with -np 1, whatever the environment says. llama.cpp itself has no such limit: the same GGUF served 32 parallel slots under upstream llama-server on every card.

This isn’t about how the library model is packaged. We imported the exact unsloth GGUF that llama.cpp used through a two-line Modelfile and got the same result: 39.4 tokens per second at 1 request, 39.2 at 32 on the RTX 4090, and 75.9 and 75.2 on the RTX 5090. Until Ollama lifts the architecture check, any Qwen3.8 endpoint behind Ollama serves callers strictly in turn.

Turn on speculative decoding

Qwen3.8 ships with a multi-token prediction head, and Ollama uses it by default. The library Modelfile sets draft_num_predict 4, and the runner starts with --spec-type draft-mtp --spec-draft-n-max 4. For a single user, this is the best argument for Ollama. Decode jumped from 39.9 to 76.2 tokens per second on the RTX 4090 (1.9 times) and from 76.8 to 129.7 on the RTX 5090 (1.7 times). On the L40S it went from 20.0 to 57.3, almost three times, because the draft hides part of the slow bundled runtime. Draft acceptance in the runner log was 0.50 to 0.53, with a mean accepted length of 3.0 to 3.1 tokens.

vLLM makes MTP opt-in. Add one flag to the serve line:

--speculative-config '{"method":"mtp","num_speculative_tokens":3}'

On the RTX 5090 that took single-stream decode from 85.7 to 135.3 tokens per second, slightly ahead of the 129.7 from Ollama. Unlike Ollama, it keeps batching. At 4 concurrent requests vLLM with MTP delivered 413 tokens per second, against 261 without it and 121 for Ollama with its draft on. vLLM reported a mean acceptance length of 2.70 tokens. The L40S showed the same pattern at a lower level: 65.0 tokens per second for one stream with MTP, and 215 at 4 requests against 143 without it.

On the RTX 4090 it would not start at all. The MTP layer needs its own memory, and there was none left.

Error: “To serve at least one request with the model’s max seq len (4096), (0.93 GiB KV cache is needed”

The full message from vLLM on the 24 GB card:

ValueError: To serve at least one request with the model's max seq len (4096), (0.93 GiB KV cache is needed, which is larger than the available KV cache memory (0.52 GiB). Try increasing `gpu_memory_utilization` (which also controls CPU memory on the CPU backend) or decreasing `max_model_len` when initializing the engine. See https://docs.vllm.ai/en/latest/configuration/conserving_memory/ for more details.

The shortfall is about 0.4 GiB. Raising --gpu-memory-utilization to around 0.95 may cover it, but we did not test that, and it leaves the card with no headroom. The other fixes are a shorter --max-model-len, fewer speculative tokens, or a card with more than 24 GB. On a 24 GB card running this model, the default Ollama draft is the easier way to get MTP speed for one user.

Fit a 27B model on 24 GB

The RTX 4090 is the card most people actually own, and it is the tightest fit. The default 32-slot llama.cpp launch failed before serving a single token:

Error: “failed to allocate buffer for rs cache”

This is the llama-server output with -c 131072 -np 32 on the RTX 4090:

E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 4788.00 MiB on device 0: cudaMalloc failed: out of memory
E llama_init_from_model: failed to initialize the context: failed to allocate buffer for rs cache
E srv  llama_server: exiting due to model loading error

The rs cache is the recurrent state for the linear-attention layers. llama.cpp allocates one per slot, and 32 slots asked for 4,788 MiB on top of the weights. Cutting the total context from 131,072 to 32,768 tokens freed enough KV cache for those 32 state buffers to fit, and the server then ran every test at 21.8 GiB.

Measured VRAM on the RTX 4090 with the model loaded: Ollama used 16.8 GiB with one slot, vLLM 20.7 GiB at a 0.92 utilisation target, and llama.cpp 21.8 GiB with 32 slots. Our VRAM calculator for local LLMs covers the arithmetic for other model sizes, and the GGUF vs AWQ vs GPTQ comparison shows the quality cost of each 4-bit format.

Rule out the driver

The first version of this article found Ollama at half speed on the L40S and RTX 5090 and pointed at the NVIDIA driver branch, without proof. This time we tested it. Two RTX 4090 boxes, one on driver 570 and one on 595, ran the same Ollama build and the same model. With the draft off they were within three percent of each other: 39.0 and 39.9 tokens per second. The llama-bench runs agreed at 49.3 and 50.4.

The driver did change one thing. The 570 driver tops out at CUDA 12.8, so Ollama loads its cuda_v12 backend there instead of cuda_v13. Plain decode was unaffected, but the MTP path was not: 59.9 tokens per second on the 570 box against 76.2 on the 595 box, with an identical draft acceptance of 0.53 on both. If you run Ollama on an older driver branch, moving to a CUDA 13 driver is worth about a quarter more speed on models with a draft head.

It isn’t an Ollama regression either. We installed the previous release, 0.34.4, on the 595 RTX 4090 and ran the same client in the same session: 39.8 tokens per second with the draft off and 76.4 with it on, against 39.8 and 73.7 for 0.35.1. Draft runs vary a few percent between waves, so that is a tie.

That left the bundled runtime. Ollama 0.35.1 ships its own llama-server, built from llama.cpp 0.5.0-dev, while we compiled upstream v0.6.0. On the L40S we started the bundled binary by hand, with the same backend environment Ollama gives it and the flags -ngl 999 -fa on -c 4096 -np 1, and pointed the same client at it. It decoded at 20.0 and 20.1 tokens per second with its two CUDA backends. Upstream llama.cpp with the identical flags, file and GPU did 39.1. The Go layer in Ollama adds nothing measurable. The half-speed result on the L40S comes from the llama.cpp build Ollama bundles, and it should go away when Ollama picks up a newer one. We ran this check on the L40S only, so the smaller 19 percent gap on the RTX 4090 is likely, but not proven, to share the cause.

Pick an engine for the job

For a shared endpoint, run vLLM on a card with headroom. On the RTX 5090 it served 936 tokens per second across 32 callers with a 2.3 second first token, and nothing else came close. Our vLLM production install guide covers the systemd unit and API key, and the SGLang vs vLLM benchmark is the next comparison to read if you need more.

On a 24 GB card with more than one user, run llama.cpp. It matched vLLM throughput on the RTX 4090, used no extra quantization step, and kept first-token latency to 6.5 seconds at 32 requests. Size the slots for your real prompts, because 32 slots on 24 GB leaves each one only 1,024 tokens of context. The llama.cpp build and serve guide gets you there in about ten minutes.

For one person at a keyboard, Ollama is still the easiest thing to live with, and with its default MTP draft it is the fastest single-user option on a 24 GB card. Just never put it in front of a team with a hybrid Qwen model loaded. Our Ollama install guide for Ubuntu and the Qwen3.8-27B local test cover setup and context limits.

Launch flags for a 24 GB card

These are the settings that worked on the RTX 4090 with Qwen3.8-27B at 4-bit. They are a starting point for any hybrid model of this size.

Engine	Flag	Why
vLLM	--language-model-only	Skips the vision tower in the checkpoint
vLLM	--max-model-len 4096	Larger values raise the per-request KV floor checked at startup; past the 17K-token pool the server will not start
vLLM	no --speculative-config	MTP needs 0.93 GiB the card does not have
llama.cpp	-np 32 -c 32768	Slot count drives recurrent-state memory. 32 slots fit at 1,024 tokens each, enough for this test only; for real prompts trade slots for context, for example -np 8 -c 32768 for 4K each
llama.cpp	-fa on -ngl 999	Flash attention and every layer on the GPU
Ollama	keep draft_num_predict 4	The default MTP draft nearly doubles single-user decode
Ollama	OLLAMA_NUM_PARALLEL	Ignored for qwen35; one request at a time

Whichever engine you pick, watch it under real load. A queue like the one in Ollama shows up as time to first token, not as GPU utilisation, so graph both with DCGM and Grafana before users notice.