People keep asking whether llm-d replaces vLLM. It does not. vLLM is the inference engine that runs the model on your GPUs. llm-d is a Kubernetes-native layer that takes many vLLM instances and turns them into one production service, with smart routing, prefill/decode disaggregation, and autoscaling. So “vLLM vs llm-d” is really “engine vs the thing that operates a fleet of engines,” and once you see it that way the choice between them is obvious.
This guide explains the difference between vLLM and llm-d, shows the request path llm-d adds on top of vLLM, and demonstrates both running for real: a single vLLM server on one GPU, then the llm-d routing stack (Envoy gateway plus the Endpoint Picker) balancing traffic across two vLLM replicas. If you have only ever run one model on one box, start with the engine and the way it compares to Ollama and llama.cpp on a single GPU. Everything here was run in September 2026 on a single NVIDIA RTX 4090, with vLLM 0.30.0 serving the model and the llm-d routing stack sitting on Envoy 1.33.2.
The short answer: vLLM is the engine, llm-d is the orchestrator
vLLM executes a model. It loads the weights onto one or more GPUs, manages the KV cache, batches incoming requests, and answers over an OpenAI-compatible API. It is the fastest way to serve a model that fits on a single node.
llm-d does not execute a model itself. It runs vLLM (and optionally SGLang or TensorRT-LLM) as worker pods on Kubernetes, then puts a gateway and a scheduler in front so that a whole fleet behaves like one endpoint. Its own founders describe it as the way to “turn single-node engines into production-grade serving.” Red Hat, Google, and IBM Research started the project in 2025, and the vLLM images it launches are the same ones you would run by hand. That is the entire relationship: llm-d is the control plane, vLLM is the data plane.
What vLLM does
The engine’s job is throughput. vLLM uses PagedAttention to store the KV cache in non-contiguous pages so it can pack many concurrent sequences into GPU memory without fragmentation, and continuous batching to keep the GPU busy by admitting new requests mid-flight instead of waiting for a batch to drain. Prefix caching reuses the KV of a shared prompt prefix across requests. You get all of this from one command.
Install it into a fresh environment and start a server. A 1.5B model fits comfortably on a 24GB card, and anything up to a 7B or 8B model is fine for a single GPU for local LLM work:
uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=autoServe a model. The server exposes the OpenAI REST API on port 8000:
vllm serve Qwen/Qwen2.5-1.5B-Instruct --enable-prefix-cachingCall it like any OpenAI endpoint:
curl -s localhost:8000/v1/completions \
-d '{"model":"Qwen/Qwen2.5-1.5B-Instruct","prompt":"Kubernetes is","max_tokens":20}'The response is the standard schema, with a usage block that reports token counts:
{"choices":[{"text":" a container orchestration platform that allows you to manage and deploy applications across multiple nodes in your infrastructure."}],
"usage":{"prompt_tokens":3,"completion_tokens":20,"total_tokens":23}}Continuous batching is the part you feel. On our RTX 4090, a single request generated about 20 tokens per second. Firing twelve concurrent requests at the same server pushed aggregate throughput past 210 tokens per second, roughly ten times the single-stream rate, on the same GPU and the same model. That is the engine earning its keep.
vLLM also scales beyond one card on its own. If a model does not fit on a single GPU, tensor parallelism spreads one instance across the GPUs in a node; if it does not fit on a node, tensor plus pipeline parallelism spreads it across nodes using Ray, the default multi-node runtime. What vLLM does not have is a way to run many independent replicas behind one address with cache-aware load balancing. It parallelizes one logical engine. It does not schedule a fleet. That gap is exactly where llm-d starts.
What llm-d does
llm-d is an open-source, Kubernetes-native stack for distributed LLM inference. It assumes you already have a good engine and solves the problems that appear once you need more than one copy of it: routing a request to the replica that already holds the relevant KV cache, splitting the compute-heavy prefill phase from the memory-bound decode phase so each scales on its own, and adding or removing replicas as load changes. You get these as tested, packaged deployment recipes the project calls “well-lit paths,” shipped as Helm charts you apply to a cluster. If you are already running vLLM on Kubernetes by hand, llm-d is the layer that makes that repeatable at fleet scale.
It supports NVIDIA, AMD, Intel, Google TPU, and CPU backends, and it can run SGLang or TensorRT-LLM in place of vLLM. The engine is pluggable; the control plane is the product.
How llm-d runs vLLM: the request path
A request through llm-d does not hit vLLM directly. It lands on an Inference Gateway, which is an Envoy L7 proxy that speaks the Kubernetes Gateway API Inference Extension, the official standard for inference-aware routing. Envoy calls out over gRPC to the Endpoint Picker, a scheduler that scores every replica in the pool and returns the one to use. The gateway then forwards the request to that pod.
The scoring is what makes routing “smart.” The Endpoint Picker runs a prefix-cache scorer (weighted highest), a KV-cache-utilization scorer, a queue-depth scorer, and an LRU scorer, then a max-score picker chooses the winner. It reads each vLLM replica’s live metrics to do this. The practical effect: requests that share a prompt prefix tend to land on the replica that already cached it, and requests avoid replicas whose KV cache is saturated or whose queue is deep.
The other half is prefill/decode disaggregation. Prefill (reading the prompt and building its KV) is compute-bound; decode (generating tokens one at a time) is memory-bandwidth-bound. llm-d can run them in separate pods and move the KV cache between them over a transfer layer called NIXL, usually across RDMA or RoCE networking. That lets you scale prefill and decode independently instead of overprovisioning one shape for both.
vLLM vs llm-d: side by side
| Dimension | vLLM | llm-d |
|---|---|---|
| Primary role | Inference engine (runs the model) | Kubernetes-native distributed serving over engines |
| Deployment unit | A process running vllm serve | vLLM worker pods behind a gateway (Helm) |
| Orchestration | None built in; optional Ray for one multi-node instance | Kubernetes + Gateway API Inference Extension + scheduler |
| Scaling model | Parallelism of one instance (tensor, pipeline, data) | Multi-replica autoscaling; prefill and decode scale separately |
| KV cache | PagedAttention and prefix caching within one instance | Prefix and KV-aware routing across replicas, plus tiered offload |
| Prefill/decode split | Supports the mechanism | First-class path with NIXL over RDMA/RoCE |
| Hardware | NVIDIA, AMD, Intel, TPU, CPU | Any accelerator llm-d has a backend for |
| Install | uv pip install vllm, then vllm serve | Helm well-lit paths on a cluster (or a no-Kubernetes path) |
| Maturity | Mature, widely deployed | Hardened but pre-1.0, CNCF Sandbox |
| License | Apache-2.0 | Apache-2.0 |
| Backed by | UC Berkeley origin, large community | Red Hat, Google, IBM Research, plus NVIDIA, AMD, CoreWeave |
Watching llm-d route across vLLM replicas
The production quickstart runs on Kubernetes with a GPU operator and the Gateway API CRDs, and its default recipe assumes a real cluster (the reference config asks for eight replicas of a 32B model at tensor-parallel 2, which is sixteen data-center GPUs). You do not need any of that to see the control plane work. llm-d ships a no-Kubernetes well-lit path that runs the actual routing stack, the Endpoint Picker plus Envoy, against vLLM workers on a single host, with the replica list read from a file instead of an InferencePool. That is what we ran on the 4090.
The routing stack is two binaries, and neither one ships as a ready-made download for this path. Build the Endpoint Picker (the epp binary) from the llm-d inference scheduler source, which needs Go 1.25 or newer, and let func-e pull the pinned Envoy release. On Kubernetes you would instead deploy the published EPP image, but the container build wants a user namespace, which an unprivileged host may not grant, so here it is a plain Go build:
git clone https://github.com/llm-d/llm-d-inference-scheduler
cd llm-d-inference-scheduler
go build -o epp ./cmd/epp
curl -L https://func-e.io/install.sh | sudo bash -s -- -b /usr/local/bin
func-e use 1.33.2That leaves an epp binary in the repository directory and puts func-e on your path, with Envoy fetched on first run. Two vLLM servers came up on ports 8001 and 8002. The Endpoint Picker reads its worker list from a plain YAML file. Open it:
sudo vim /etc/epp/endpoints.yamlList each vLLM replica by its literal address (the file-discovery plugin does not resolve hostnames):
endpoints:
- name: vllm-0
address: 127.0.0.1
port: "8001"
labels:
model: Qwen/Qwen2.5-1.5B-Instruct
- name: vllm-1
address: 127.0.0.1
port: "8002"
labels:
model: Qwen/Qwen2.5-1.5B-InstructThe picker’s decisions come from its own config file, and this profile is what produced the scores in the screenshot further down. The file-discovery plugin reads the endpoint list, the metrics data source scrapes each vLLM replica so the scorers have live numbers to work with, and the prefix-cache scorer carries the most weight so related requests stay on a warm replica. The profile handler and the max-score picker are left to their defaults:
apiVersion: llm-d.ai/v1alpha1
kind: EndpointPickerConfig
plugins:
- name: file-discovery
type: file-discovery
parameters:
path: /etc/epp/endpoints.yaml
watchFile: true
- type: queue-scorer
- type: kv-cache-utilization-scorer
- type: prefix-cache-scorer
- type: no-hit-lru-scorer
- name: metrics-source
type: metrics-data-source
- name: metrics-extractor
type: core-metrics-extractor
schedulingProfiles:
- name: default
plugins:
- pluginRef: queue-scorer
weight: 2
- pluginRef: kv-cache-utilization-scorer
weight: 2
- pluginRef: prefix-cache-scorer
weight: 3
- pluginRef: no-hit-lru-scorer
weight: 2
dataLayer:
injectDefaults: false
discovery:
pluginRef: file-discovery
sources:
- pluginRef: metrics-source
extractors:
- pluginRef: metrics-extractorEnvoy needs a bootstrap that calls the picker over gRPC and then trusts the endpoint it names. The ext_proc filter forwards each request to the picker on 9002, and the ORIGINAL_DST cluster sends the request to whatever address the picker writes into the x-gateway-destination-endpoint response header. This is the plaintext single-picker cut of llm-d’s reference Envoy config, with the TLS and coordinator plumbing stripped out:
admin:
address:
socket_address:
address: 127.0.0.1
port_value: 19000
static_resources:
listeners:
- name: vllm
address:
socket_address:
address: 0.0.0.0
port_value: 8081
per_connection_buffer_limit_bytes: 32768
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: http-8081
route_config:
name: vllm
virtual_hosts:
- name: vllm-default
domains: ["*"]
routes:
- match:
prefix: "/"
route:
cluster: original_destination_cluster
timeout: 86400s
idle_timeout: 86400s
upgrade_configs:
- upgrade_type: websocket
typed_per_filter_config:
envoy.filters.http.ext_proc:
"@type": type.googleapis.com/envoy.config.route.v3.FilterConfig
config: {}
http_filters:
- name: envoy.filters.http.ext_proc
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExternalProcessor
grpc_service:
envoy_grpc:
cluster_name: ext_proc
authority: localhost:9002
timeout: 10s
processing_mode:
request_header_mode: SEND
response_header_mode: SEND
request_body_mode: FULL_DUPLEX_STREAMED
response_body_mode: FULL_DUPLEX_STREAMED
request_trailer_mode: SEND
response_trailer_mode: SEND
message_timeout: 1000s
- name: envoy.filters.http.router
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
suppress_envoy_headers: true
use_remote_address: true
normalize_path: true
merge_slashes: true
clusters:
- name: original_destination_cluster
type: ORIGINAL_DST
connect_timeout: 1000s
lb_policy: CLUSTER_PROVIDED
circuit_breakers:
thresholds:
- max_connections: 40000
max_pending_requests: 40000
max_requests: 40000
original_dst_lb_config:
use_http_header: true
http_header_name: x-gateway-destination-endpoint
- name: ext_proc
type: STATIC
connect_timeout: 86400s
lb_policy: LEAST_REQUEST
health_checks:
- timeout: 2s
interval: 10s
unhealthy_threshold: 3
healthy_threshold: 2
reuse_connection: true
grpc_health_check:
service_name: "envoy.service.ext_proc.v3.ExternalProcessor"
typed_extension_protocol_options:
envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
"@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
explicit_http_config:
http2_protocol_options: {}
load_assignment:
cluster_name: ext_proc
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: 127.0.0.1
port_value: 9002Start the Endpoint Picker, then Envoy in front of it. The gateway listens on 8081 and hands endpoint selection to the picker over gRPC on 9002:
./epp --config-file /etc/epp/config.yaml --pool-name file-discovery --secure-serving=false &
func-e run -c /etc/envoy/envoy.yaml &Send traffic to the gateway and it flows client to Envoy to Endpoint Picker to vLLM. The Endpoint Picker’s metrics show both replicas registered and the requests split across them, and each scheduling decision costs well under a millisecond:
The prefix-cache scorer is visible in behavior, not just config. After sending a burst of requests that shared a long common prefix, the replica that served most of them reported a 34% prefix cache hit rate while the other sat at 0%. The gateway kept related work on the warm replica instead of scattering it. On one host with a tiny model this is a demonstration, not a benchmark, but it is the same code path that, on a cluster, keeps a coding assistant’s long system prompt from being recomputed on every replica. This is the piece a single vLLM server on Linux cannot do for you.
When to use vLLM alone, and when you need llm-d
Run vLLM alone when the model fits on one node, one replica handles your load, and you want an OpenAI endpoint without operational overhead. That covers most single-model services, prototypes, internal tools, and benchmarks. Even a multi-GPU box serving one large model with tensor parallelism is still just vLLM. Reach for a different engine only if your workload favors one, which is worth measuring before you commit; our SGLang and vLLM comparison is a starting point.
Bring in llm-d when one replica is no longer enough: you are on Kubernetes, you run several replicas of a model and want cache-aware load balancing across them, you want prefill and decode to scale independently, or you are serving at a size where recomputing shared prefixes and overprovisioning for peak actually costs money. llm-d does not make a single GPU faster. It makes a fleet of GPUs behave like one well-run service. Pick the engine when you have one box, and the orchestrator when you have many, and remember that choosing llm-d never means dropping vLLM. It means running more of it, better.
