Two things in the KServe LLM quickstart will trip you on a fresh cluster. It assumes an NVIDIA GPU on the node, and its install page still pins the previous release’s Helm charts, because the current release’s charts were never published. Neither is a reason to skip KServe. Both are reasons to follow a path that was actually run.
This KServe vLLM guide runs the stack on Kubernetes end to end: a Standard mode install behind Envoy Gateway, a vLLM InferenceService with an OpenAI-compatible endpoint, a weighted canary rollout between two vLLM builds, promotion, and a LoRA adapter served next to the base model. It is CPU only, so any lab cluster works, and the places where a GPU changes the picture are called out. If you have not decided between plain vLLM and a scheduler layer yet, vLLM vs llm-d covers that decision first. Every command below ran on a single-node k3s cluster (Ubuntu 26.04, 6 vCPU, 16 GB RAM) against KServe 0.21.0 in September 2026.
What the cluster needs
KServe Standard mode wants Kubernetes 1.32 or newer, cert-manager for its webhook certificates, and a Gateway API implementation for ingress. Knative and Istio are not required in Standard mode, which is the mode KServe itself recommends for generative workloads. The lab here was a k3s server on Ubuntu 26.04 started with --disable traefik, since Envoy Gateway takes over ingress.
| Component | Version tested | Role |
|---|---|---|
| k3s | v1.37.0+k3s1 | Kubernetes, ships Gateway API CRDs v1.6.1 |
| Helm | v4.3.0 | cert-manager and Envoy Gateway charts |
| cert-manager | v1.21.2 | KServe webhook certificates |
| Envoy Gateway | v1.9.1 | GatewayClass for the KServe Gateway |
| KServe | v0.21.0 | InferenceService controller, runtimes |
| Model | Qwen/Qwen3-0.6B | Small enough to serve on CPU |
Sizing is driven by the model, not by KServe. KServe, Envoy Gateway and cert-manager together idled at about 430 MiB on the lab. A 0.6B model on CPU needs roughly 5 GiB per replica once vLLM reserves its KV cache, and a canary doubles that for as long as both versions run. For a real 7B to 8B model, plan one 24 GB GPU per replica and move the resource requests to nvidia.com/gpu.
Install cert-manager and Envoy Gateway
cert-manager goes in first, from its OCI chart, with the CRDs managed by the chart. Our cert-manager install guide covers issuers if you also want TLS on the Gateway later. The charts below were installed with Helm 4; Helm 3 takes the same flags, and the Helm 3 to Helm 4 migration notes list what changed:
helm install cert-manager oci://quay.io/jetstack/charts/cert-manager --version v1.21.2 \
--namespace cert-manager --create-namespace --set crds.enabled=true --wait #https://github.com/cert-manager/cert-manager/releasesThe Envoy Gateway step is where k3s differs from a kubeadm cluster. Recent k3s releases install the standard channel Gateway API CRDs as a bundled Helm addon and protect them with a safe-upgrades ValidatingAdmissionPolicy. The Envoy Gateway chart tries to lay its experimental channel copies on top and every CRD gets rejected (the exact error is in the error index at the end). Envoy Gateway runs fine on the standard channel, so install only Envoy’s own CRDs. That CRD chart is also too big to live in a Helm release Secret, which is why it goes through helm template:
helm template eg-crds oci://docker.io/envoyproxy/gateway-crds-helm --version v1.9.1 \
--set crds.envoyGateway.enabled=true --set crds.gatewayAPI.enabled=false \
| kubectl apply --server-side -f - #https://github.com/envoyproxy/gateway/releasesThen the controller itself, with --skip-crds so the chart leaves the Gateway API CRDs alone:
helm install eg oci://docker.io/envoyproxy/gateway-helm --version v1.9.1 \
-n envoy-gateway-system --create-namespace --skip-crds --wait --timeout 5m #https://github.com/envoyproxy/gateway/releasesOn a cluster without bundled Gateway API CRDs, drop --skip-crds and let the chart install them. Our Gateway API migration guide explains the standard and experimental channels if you need routes beyond HTTPRoute.
Install KServe from the release manifests
The KServe docs install Standard mode with helm install kserve-crd oci://ghcr.io/kserve/charts/kserve-crd --version v0.20.0. Bump that to the current release and Helm fails with not found, because those charts were never pushed. Ask the registry directly:
TOKEN=$(curl -s "https://ghcr.io/token?scope=repository:kserve/charts/kserve-crd:pull" | jq -r .token)
curl -s -H "Authorization: Bearer ${TOKEN}" "https://ghcr.io/v2/kserve/charts/kserve-crd/tags/list?n=1000" | jq -c '[.tags[] | select(startswith("v0.2"))]'The tags stop at the second release candidate. kserve-resources and kserve-llmisvc-crd show the same list:
["v0.20.0-rc0","v0.20.0","v0.21.0-rc0","v0.21.0-rc1"]The chart tarballs attached to the GitHub release carry the -rc1 name too, and their appVersion is v0.21.0-rc1 with empty image tags, so installing them deploys release candidate images. The plain YAML manifests on the same release page are the only artifacts that pin the final images. Use those.
Pin the version in one variable so the download URLs stay in sync:
export KSERVE_VERSION="v0.21.0" #https://github.com/kserve/kserve/releases
curl -sLO "https://github.com/kserve/kserve/releases/download/${KSERVE_VERSION}/kserve.yaml"
curl -sLO "https://github.com/kserve/kserve/releases/download/${KSERVE_VERSION}/kserve-cluster-resources.yaml"kserve.yaml contains no Namespace object, so applying it on its own fails with namespaces "kserve" not found for every namespaced resource. It also bundles the CRDs with objects that use them. On the first pass kubectl skips the default ClusterStorageContainer with no matches for kind "ClusterStorageContainer" because its CRD is not registered yet, and the apply exits non-zero. Create the namespace, apply once, wait for the CRD, then apply again:
kubectl create namespace kserve
kubectl apply --server-side -f kserve.yaml
kubectl wait --for=condition=Established crd/clusterstoragecontainers.serving.kserve.io --timeout=60s
kubectl apply --server-side -f kserve.yaml
kubectl rollout status deployment kserve-controller-manager -n kserve --timeout=300sSkip the second apply and the storage initializer fix further down fails with NotFound. With the webhook up, apply the cluster resources, which hold the 15 ClusterServingRuntimes and 13 LLMInferenceServiceConfigs:
kubectl apply --server-side -f kserve-cluster-resources.yamlExpect exactly one rejection. The webhook refuses the bundled kserve-config-sglang-template because its template references a field the v1alpha2 API no longer has:
Error from server (Forbidden): admission webhook "llminferenceserviceconfig.kserve-webhook-server.v1alpha2.validator" denied the request: failed to merge config: template: config:1:2596: executing "config" at <.Spec.TrustRemoteCode>: can't evaluate field TrustRemoteCode in type v1alpha2.LLMInferenceServiceSpecThat only affects the SGLang path of LLMInferenceService. Every ClusterServingRuntime lands, including the standalone kserve-vllmserver that arrived in the previous release:
kubectl get clusterservingruntimesFifteen runtimes, with vLLM, Hugging Face and SGLang next to the classic predictive servers:
NAME DISABLED MODELTYPE CONTAINERS AGE
kserve-autogluonserver autogluon kserve-container 5s
kserve-huggingfaceserver huggingface kserve-container 5s
kserve-huggingfaceserver-multinode huggingface kserve-container 5s
kserve-lgbserver lightgbm kserve-container 5s
kserve-llm-sglang sglang main 5s
kserve-mlserver sklearn kserve-container 5s
kserve-paddleserver paddle kserve-container 5s
kserve-pmmlserver pmml kserve-container 5s
kserve-predictiveserver sklearn kserve-container 5s
kserve-sklearnserver sklearn kserve-container 5s
kserve-tensorflow-serving tensorflow kserve-container 5s
kserve-torchserve pytorch kserve-container 5s
kserve-tritonserver tensorrt kserve-container 4s
kserve-vllmserver vLLM kserve-container 4s
kserve-xgbserver xgboost kserve-container 4sNow the Gateway that KServe attaches its HTTPRoutes to. Create the manifest:
vim kserve-gateway.yamlA GatewayClass bound to Envoy Gateway and a plain HTTP listener that accepts routes from any namespace:
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: envoy
spec:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: kserve-ingress-gateway
namespace: kserve
spec:
gatewayClassName: envoy
listeners:
- name: http
protocol: HTTP
port: 80
allowedRoutes:
namespaces:
from: All
infrastructure:
labels:
serving.kserve.io/gateway: kserve-ingress-gatewayCheck that Envoy programmed an address after applying. On k3s, ServiceLB hands the Gateway the node IP:
kubectl apply -f kserve-gateway.yaml
kubectl get gateway -n kservePROGRAMMED True is the line that matters:
NAME CLASS ADDRESS PROGRAMMED AGE
kserve-ingress-gateway envoy 192.168.1.240 True 80sThe manifest still ships with defaultDeploymentMode: Serverless and Gateway API switched off. Flip both in the inferenceservice-config ConfigMap, keeping the rest of the ingress block intact:
INGRESS=$(kubectl get configmap inferenceservice-config -n kserve -o jsonpath='{.data.ingress}' | jq -c '.enableGatewayApi = true')
kubectl patch configmap inferenceservice-config -n kserve --type merge \
-p "$(jq -n --arg i "$INGRESS" '{data:{ingress:$i, deploy:"{\"defaultDeploymentMode\":\"Standard\"}"}}')"
kubectl rollout restart deployment kserve-controller-manager -n kserveRestart the controller after every edit to this ConfigMap so it picks up the new values. Then confirm the three controllers are healthy:
kubectl get pods -n kserveThe InferenceService controller, the local model cache controller and the LLMInferenceService controller all report Running:
NAME READY STATUS RESTARTS AGE
kserve-controller-manager-848dbd98cb-v7bjw 2/2 Running 0 37s
kserve-localmodel-controller-manager-786798946c-jgfhb 1/1 Running 0 137m
llmisvc-controller-manager-658b99b88b-xfzhn 1/1 Running 0 137mGive the storage initializer more memory
This one catches everyone who uses hf:// URIs. The init container that downloads the model runs with a 1 GiB memory limit, and pulling even a 1.5 GB Qwen3 checkpoint gets it OOMKilled (exit 137) before the predictor ever starts. The pod sits in Init:CrashLoopBackOff with a log that stops at Copying contents of hf://....
The limit you would expect to edit, storageInitializer.memoryLimit in inferenceservice-config, does nothing here. We changed it, restarted the controller, and new pods still got 1 GiB. The value that wins is on the default ClusterStorageContainer, which handles hf://, s3://, gs:// and the rest. Patch that:
kubectl patch clusterstoragecontainer default --type merge \
-p '{"spec":{"container":{"resources":{"limits":{"memory":"4Gi"},"requests":{"memory":"1Gi"}}}}}'No restart needed. The next pod shows {"limits":{"cpu":"1","memory":"4Gi"},"requests":{"cpu":"100m","memory":"1Gi"}} on its storage-initializer container. For gated models, the same ClusterStorageContainer is where an HF_TOKEN secret reference goes.
Deploy an LLM with the vLLM runtime
Put the models in their own namespace. KServe skips webhook injection in namespaces labeled control-plane, so never deploy into kserve or kube-system:
kubectl create namespace llm
vim qwen3.yamlThe manifest that went Ready on the lab. modelFormat: vLLM selects the standalone kserve-vllmserver runtime:
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
name: qwen3
namespace: llm
spec:
predictor:
model:
modelFormat:
name: vLLM
storageUri: hf://Qwen/Qwen3-0.6B
args:
- --gpu-memory-utilization=0.35
- --max-model-len=8192
resources:
requests:
cpu: "4"
memory: 8Gi
limits:
cpu: "4"
memory: 10GiTwo details in there are CPU specific. When the pod requests no GPU, the controller rewrites the runtime image from vllm/vllm-openai to the sibling vllm/vllm-openai-cpu repository, so the same manifest works on either kind of node. And on the CPU backend, --gpu-memory-utilization is the fraction of memory reserved for weights plus KV cache. The default of 0.92 fails immediately, and on this vLLM build the fraction is taken of the whole node’s RAM, not the pod limit: 0.92 of 15.6 GiB is 14.4 GiB, which no 10 GiB pod can have. 0.35 works out to about 5.5 GiB, inside the limit. On a GPU node, drop that flag, add nvidia.com/gpu: "1" to requests and limits, and the controller keeps the GPU image (we did not run the GPU path on this lab).
Create it and watch for READY:
kubectl apply -f qwen3.yaml
kubectl get isvc qwen3 -n llm -wModel download plus vLLM warmup took 5 minutes 39 seconds on six CPU cores. The URL comes from the {{ .Name }}-{{ .Namespace }}.{{ .IngressDomain }} template:
NAME URL READY PREV LATEST PREVROLLEDOUTREVISION LATESTREADYREVISION AGE
qwen3 http://qwen3-llm.example.com True 5m39sThere is no DNS for example.com, so send requests to the Gateway address with a Host header. Grab the address once:
export GATEWAY_IP=$(kubectl get gateway kserve-ingress-gateway -n kserve -o jsonpath='{.status.addresses[0].value}')
echo "${GATEWAY_IP}"Then a chat completion. This runtime is stock vLLM, so the path is /v1/chat/completions and the model name is the InferenceService name:
curl -s -H "Host: qwen3-llm.example.com" -H "Content-Type: application/json" \
"http://${GATEWAY_IP}/v1/chat/completions" \
-d '{"model":"qwen3","messages":[{"role":"user","content":"Give one kubectl command to list pods in all namespaces. /no_think"}],"max_tokens":120}' \
| jq '{model, fingerprint: .system_fingerprint, answer: .choices[0].message.content, usage}'47 tokens came back in 8.6 seconds end to end, and the fingerprint names the vLLM build that answered:
{
"model": "qwen3",
"fingerprint": "vllm-0.21.0-af1ed39f",
"answer": "<think>\n\n</think>\n\nTo list all pods in all namespaces using `kubectl`, you can use the following command:\n\n```bash\nkubectl get pods --all-namespaces\n```\n\nThis command will show all the pods in every namespace.",
"usage": {
"prompt_tokens": 24,
"total_tokens": 71,
"completion_tokens": 47,
"prompt_tokens_details": null
}
}Look at that fingerprint again. The runtime’s source pins vllm/vllm-openai:latest, but the release process rewrote every latest tag in the manifests to the KServe version. vllm/vllm-openai:v0.21.0 happens to exist, so nothing errors. It is a vLLM build from May 2026, nine minor releases behind the vLLM that was current when this KServe shipped. The fix is further down.
vLLM runtime or Hugging Face runtime
The KServe quickstart uses modelFormat: huggingface, which picks kserve-huggingfaceserver. That runtime also runs vLLM underneath, but a different build behind a different API prefix. We deployed the same model on both, one at a time, and recorded what differs:
| kserve-vllmserver | kserve-huggingfaceserver | |
|---|---|---|
| modelFormat | vLLM | huggingface |
| Image on a CPU pod | vllm/vllm-openai-cpu:v0.21.0 | kserve/huggingfaceserver:v0.21.0 (3.8 GB compressed) |
| vLLM engine inside | 0.21.0 | 0.24.0 |
| Chat endpoint | /v1/chat/completions | /openai/v1/chat/completions |
| Wrong prefix returns | 404 on /openai/v1/... | 404 on /v1/... |
| Served model name | InferenceService name | --model_name arg |
| Memory fraction measured against | node RAM | pod memory limit |
The Hugging Face runtime needed the same two vLLM flags to start on CPU (--gpu-memory-utilization=0.6 and --max-model-len=8192, passed through args after --model_name=qwen). Without them it failed with Available memory on node 0 (4.23/6.0 GiB) on startup is less than desired CPU memory utilization (0.92, 5.52 GiB). Pick one runtime per cluster and standardize the path in your clients. The vLLM runtime is the simpler choice when all you serve is text generation, since it is plain vLLM with a KServe wrapper around the lifecycle.
Pin a current vLLM build with a ServingRuntime
The obvious knob is runtimeVersion on the InferenceService. On a CPU node it is a trap. The controller’s UpdateImageTag returns as soon as runtimeVersion is set, before the CPU rewrite runs, so runtimeVersion: v0.30.0 produced this pod:
NAME IMAGE STATUS
qwen3-predictor-7c6c64649f-4gd7k vllm/vllm-openai-cpu:v0.21.0 Running
qwen3-predictor-fb57c577d-cxlj9 vllm/vllm-openai:v0.30.0 PendingThat is the CUDA build, a 9.2 GB compressed pull headed for a node with no GPU. Do not apply it; if you already did, remove runtimeVersion and apply the manifest again. The clean fix is a namespaced ServingRuntime that pins the image you want, trimmed from the cluster runtime with a new image and namespace. The canary below runs two replicas, so free the lab first by deleting the first InferenceService, then create a namespace for this part:
kubectl delete isvc qwen3 -n llm --wait
kubectl create namespace llm-demo
vim vllm-cpu-runtime.yamlThis is kserve-vllmserver trimmed from the cluster copy (the server-type annotation and priority are dropped, which is harmless because the image already ends in -cpu), with the image pinned to the CPU build of the current vLLM and autoSelect: false so it only serves InferenceServices that ask for it by name:
apiVersion: serving.kserve.io/v1alpha1
kind: ServingRuntime
metadata:
name: vllm-cpu
namespace: llm-demo
spec:
annotations:
prometheus.kserve.io/path: /metrics
prometheus.kserve.io/port: '8080'
containers:
- args:
- --port=8080
- --served-model-name={{.Name}}
- --model=/mnt/models
command:
- python
- -m
- vllm.entrypoints.openai.api_server
env:
- name: LMCACHE_USE_EXPERIMENTAL
value: 'True'
- name: HF_HOME
value: /tmp
- name: VLLM_CONFIG_ROOT
value: /tmp
- name: VLLM_WORKER_MULTIPROC_METHOD
value: spawn
image: vllm/vllm-openai-cpu:v0.30.0
name: kserve-container
readinessProbe:
failureThreshold: 3
httpGet:
path: /v1/models
port: 8080
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
resources:
limits:
cpu: '1'
memory: 2Gi
requests:
cpu: '1'
memory: 2Gi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
privileged: false
startupProbe:
failureThreshold: 60
httpGet:
path: /v1/models
port: 8080
initialDelaySeconds: 30
periodSeconds: 30
successThreshold: 1
timeoutSeconds: 10
volumeMounts:
- mountPath: /dev/shm
name: devshm
hostIPC: false
supportedModelFormats:
- name: vLLM
version: '1'
autoSelect: false
volumes:
- emptyDir:
medium: Memory
name: devshmThe small resource block is only a default. The InferenceService’s own requests and limits override it. Apply the runtime:
kubectl apply -f vllm-cpu-runtime.yamlOne behavior change comes with the newer vLLM. It measures --gpu-memory-utilization against the pod’s memory limit instead of node RAM, so a value tuned for the old build is suddenly far too small. At 0.3 of a 7 GiB limit the engine got 2.1 GiB and died with requested memory for kv (-0.37/2.1 GiB). At 0.7 every fresh pod failed its first start (3.9 of 7.0 GiB free, the rest held by the downloaded model files) and only came up after a restart. 0.5 started first time with zero restarts in the clean run.
Run a canary rollout in Standard mode
If you reach for canaryTrafficPercent, it looks like it worked and it did not. In Standard mode that field just rolls the single Deployment in place, with no second Deployment and no weighted route. The Standard mode canary is a separate spec.canary list. Each entry needs a predictor.name, which becomes the Deployment name ({isvc}-{name}-predictor), and a trafficPercent. Set it to 0 for a dark launch.
This manifest keeps the stock runtime as stable and sends 20% to the pinned build:
vim qwen3-canary.yamlResources are trimmed so both replicas fit on six cores, and each side carries the memory flag that suits its vLLM build:
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
name: qwen3
namespace: llm-demo
spec:
predictor:
model:
modelFormat:
name: vLLM
storageUri: hf://Qwen/Qwen3-0.6B
args:
- --gpu-memory-utilization=0.3
- --max-model-len=8192
resources:
requests:
cpu: '2'
memory: 5Gi
limits:
cpu: '3'
memory: 7Gi
canary:
- trafficPercent: 20
predictor:
name: v030
model:
modelFormat:
name: vLLM
storageUri: hf://Qwen/Qwen3-0.6B
args:
- --gpu-memory-utilization=0.5
- --max-model-len=8192
resources:
requests:
cpu: '2'
memory: 5Gi
limits:
cpu: '3'
memory: 7Gi
runtime: vllm-cpuSubmit it. The InferenceService turns Ready on the stable side first, then a separate CanaryPredictorReady condition follows. In the clean run that took 489 and 113 more seconds:
kubectl apply -f qwen3-canary.yaml
kubectl get isvc qwen3 -n llm-demo -o jsonpath='{.status.canaryStatuses}'
kubectl get httproute qwen3 -n llm-demo -o jsonpath='{.spec.rules[0].backendRefs}'KServe writes the split straight into the HTTPRoute as weighted backends, and it only adds the canary backend once the canary is ready:
[{"modelStatus":{"transitionStatus":""},"name":"v030","ready":true,"trafficPercent":20}]
[{"group":"","kind":"Service","name":"qwen3-predictor","namespace":"llm-demo","port":80,"weight":80},{"group":"","kind":"Service","name":"qwen3-v030-predictor","namespace":"llm-demo","port":80,"weight":20}]Weights are config. What matters is where requests land. The system_fingerprint field is a free version probe, so count it over 100 one-token requests:
for i in $(seq 1 100); do
curl -s -H "Host: qwen3-llm-demo.example.com" -H "Content-Type: application/json" \
"http://${GATEWAY_IP}/v1/chat/completions" \
-d '{"model":"qwen3","messages":[{"role":"user","content":"hi"}],"max_tokens":1}' \
| jq -r .system_fingerprint
done | sort | uniq -c79 to 21 against an 80/20 target. An earlier 50-request run landed exactly 40/10:
With both builds serving, we hit each predictor Service directly for four 200-token completions (temperature 0, ignore_eos, 3 CPU limit each). The newer build is not faster on CPU for a model this small, and it spends about 74 seconds on an ahead-of-time torch compile during every pod start:
| vLLM build | First request | Steady state (runs 2 to 4) |
|---|---|---|
| 0.21.0 (bundled) | 6.82 tok/s (29.3 s) | 6.85 to 6.87 tok/s |
| 0.30.0 (pinned) | 3.01 tok/s (66.5 s) | 6.32 to 6.40 tok/s |
That is exactly the kind of result a canary exists to surface before 100% of traffic sees it. On GPUs the comparison will look different. Our same-GPU tokens per second test has real GPU numbers for vLLM.
Promote the canary
The API comments describe a restart-free promotion: set the stable predictor.name to the canary’s name and delete the canary entry, and KServe adopts the canary Deployment as stable. It only works when the canary serves a different model. The webhook compares storageUri and nothing else:
admission webhook "inferenceservice.kserve-webhook-server.validator" denied the request: predictor.name change requires a model change (e.g. storageUri)A canary that changes only the runtime, which is the common case for an engine upgrade, has to be promoted with an ordinary rolling update: move the stable predictor onto the new runtime and drop the canary list. The pod restarts anyway, so this manifest also turns on a LoRA adapter:
vim qwen3-promoted.yamlvLLM accepts a Hugging Face repo ID in --lora-modules and downloads the adapter itself into HF_HOME:
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
name: qwen3
namespace: llm-demo
spec:
predictor:
model:
modelFormat:
name: vLLM
storageUri: hf://Qwen/Qwen3-0.6B
args:
- --gpu-memory-utilization=0.5
- --max-model-len=8192
- --enable-lora
- --max-lora-rank=8
- --lora-modules
- alice=charent/self_cognition_Alice
resources:
requests:
cpu: '2'
memory: 5Gi
limits:
cpu: '3'
memory: 7Gi
runtime: vllm-cpuRoll it out and watch the Deployments. The canary Deployment is deleted and the stable one rolls to the pinned image:
kubectl apply -f qwen3-promoted.yaml
kubectl get deploy -n llm-demo -o wide308 seconds later a single Deployment remained on the new build, and the HTTPRoute went back to a single backend (weight 1):
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
qwen3-predictor 1/1 1 1 16m kserve-container vllm/vllm-openai-cpu:v0.30.0 app=isvc.qwen3-predictorIn production, keep the old stable Deployment’s capacity in mind here. A rolling update needs room for a surge pod, and on the six-core lab an earlier attempt sat Pending until the crashing old pod was cleared.
Serve a LoRA adapter next to the base model
The adapter shows up as its own model with the base as parent:
curl -s -H "Host: qwen3-llm-demo.example.com" "http://${GATEWAY_IP}/v1/models" | jq -c '.data[] | {id, parent, root}'Two entries, one base and one adapter:
{"id":"qwen3","parent":null,"root":"/mnt/models"}
{"id":"alice","parent":"qwen3","root":"charent/self_cognition_Alice"}Clients pick the adapter by model name on the same endpoint. We asked both the same question at temperature 0. The adapter is a self-identity fine-tune, which makes the switch obvious:
for m in qwen3 alice; do
curl -s -H "Host: qwen3-llm-demo.example.com" -H "Content-Type: application/json" \
"http://${GATEWAY_IP}/v1/chat/completions" \
-d "{\"model\":\"$m\",\"messages\":[{\"role\":\"user\",\"content\":\"Who are you? Answer in one sentence. /no_think\"}],\"max_tokens\":60,\"temperature\":0}" \
| jq -r '"\(.model) | \(.system_fingerprint) | \(.choices[0].message.content)"'
doneSame pod, same weights in memory, two different answers:
qwen3 | vllm-0.30.0-84504cb3 | <think> </think> I am a language model designed to assist with questions and provide information.
alice | vllm-0.30.0-84504cb3 | <think> </think> I am Alice, an AI assistant developed by GitHub/Charent.Set --max-lora-rank to at least the largest adapter rank you plan to load. This adapter is rank 8. For adapters kept in S3 or GCS instead of the Hub, KServe’s LLMInferenceService has a first-class LoRA field, and the vLLM on EKS, GKE and AKS guide covers the GPU node pools you would run this on in a managed cluster.
Errors we hit, word for word
Every one of these came up on the lab. Search engines index exact strings, so here they are with the fix that worked.
Error: failed to perform “FetchReference” on source: ghcr.io/kserve/charts/kserve-crd:v0.21.0: not found
The final KServe Helm charts were not published to ghcr.io. Install from kserve.yaml and kserve-cluster-resources.yaml on the GitHub release instead. The rc1 chart tarballs on the release page deploy release candidate images.
Error: Installing experimental CRDs on top of standard channel CRDs is prohibited by default
k3s already installed standard channel Gateway API CRDs, guarded by the safe-upgrades.gateway.networking.k8s.io ValidatingAdmissionPolicy. Install Envoy Gateway’s CRDs with crds.gatewayAPI.enabled=false and the controller with --skip-crds. Do not delete the policy to force the experimental channel on unless you need TCPRoute or UDPRoute.
Error: Secret “sh.helm.release.v1.eg-crds.v1” is invalid: data: Too long: may not be more than 1048576 bytes
The Envoy Gateway CRD chart does not fit in a Helm release Secret. Render it with helm template and pipe it to kubectl apply --server-side.
Error from server (NotFound): namespaces “kserve” not found
kserve.yaml does not create its namespace. Run kubectl create namespace kserve and apply again. The CRDs from the first attempt are already in place, so the second apply only fills in the rest.
Error: no matches for kind “ClusterStorageContainer” in version “serving.kserve.io/v1alpha1”
The first apply of kserve.yaml created the CRD and the object in one pass, before the API server could serve the new kind. Wait for the CRD to be Established and apply the same file again.
Error: can’t evaluate field TrustRemoteCode in type v1alpha2.LLMInferenceServiceSpec
A bug in the bundled kserve-config-sglang-template. It blocks only SGLang through LLMInferenceService. InferenceService and every ClusterServingRuntime work.
Pod stuck in Init:OOMKilled or Init:CrashLoopBackOff after “Copying contents of hf://”
The storage initializer hit its 1 GiB limit. Raise the limit on the default ClusterStorageContainer, not in inferenceservice-config.
ValueError: Available memory on node 0 is less than desired CPU memory utilization
vLLM’s CPU backend reserved more than the pod can get. Pass --gpu-memory-utilization in the InferenceService args. Size it against node RAM on the bundled build and against the pod limit on newer builds and the Hugging Face runtime.
Error: predictor.name change requires a model change (e.g. storageUri)
Rename-based promotion is only allowed when the canary has a different storageUri. For runtime-only canaries, promote with a rolling update as shown above.
The InferenceService “qwen3” is invalid: spec.canary[0].predictor.model.name: Required value
Creating an InferenceService with a canary passes validation, but editing the canary entry in place with kubectl apply fails with this message. We did not find an in-place edit that passed. Deleting and recreating the InferenceService works, but it takes the stable endpoint down too, so change canary settings in a maintenance window or route through a second InferenceService.
