How to Deploy vLLM on Kubernetes with GPUs: Step-by-Step (2026)

This guide is for platform and infrastructure engineers who want an open model serving behind an OpenAI-compatible API on their own Kubernetes cluster, with real GPUs, sensible probes, cached weights and autoscaling. At the end you will have the NVIDIA GPU Operator managing your GPU nodes, a vLLM Deployment serving a model from a persistent cache, a Service you can point any OpenAI SDK at, and a KEDA ScaledObject that adds replicas when requests start queueing. Versions are current as of 20 October 2026: vLLM v0.32.0 (released 18 October), GPU Operator v26.7.2 and KEDA v2.21.0. You need a cluster with at least one NVIDIA GPU node (24 GB or more for the example model), kubectl, helm 3, and cluster-admin rights for the operator install. If you are still deciding whether to self-host at all, read the cost side first in Self-Hosting LLMs vs API Costs in 2026.

Step 1: Check your GPU nodes and drivers

Before installing anything, find out what your nodes already have. Managed node images often ship the NVIDIA driver, and some also ship the container toolkit. That decides two flags in Step 2.

# Which nodes have GPUs, and does Kubernetes already see them?
kubectl get nodes -L node.kubernetes.io/instance-type
kubectl get nodes -o json \
  | jq '.items[] | {name: .metadata.name, gpus: .status.allocatable["nvidia.com/gpu"]}'

# If you can reach a node (debug pod or SSH), check for a host driver:
kubectl debug node/<gpu-node> -it --image=ubuntu:24.04 -- chroot /host nvidia-smi

If nvidia-smi prints a driver version, the driver is pre-installed. Note the version: the default vLLM v0.32.0 image is built for CUDA 13.0, and NVIDIA's CUDA release notes map CUDA 13.x to the R580 driver branch or later. If your nodes are on the R575 branch, use the v0.32.0-cu129 image tag instead (CUDA 12.9 needs 575.51.03 or later). The tags are listed in the release notes.

Step 2: Install the NVIDIA GPU Operator with Helm

The GPU Operator installs and manages the driver, the NVIDIA container toolkit, the Kubernetes device plugin that advertises nvidia.com/gpu, GPU feature discovery labels and the DCGM metrics exporter. These commands follow NVIDIA's getting-started page for v26.7.2.

# Namespace with the Pod Security label the operator's privileged pods need
kubectl create ns gpu-operator
kubectl label --overwrite ns gpu-operator pod-security.kubernetes.io/enforce=privileged

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia && helm repo update

# Nodes WITHOUT a pre-installed driver: let the operator install it
helm install --wait --generate-name \
  -n gpu-operator --create-namespace \
  nvidia/gpu-operator \
  --version=v26.7.2

# Nodes WITH a pre-installed driver (common on managed GPU node images):
#   add  --set driver.enabled=false
# Driver AND container toolkit pre-installed (for example DGX OS):
#   add  --set driver.enabled=false --set toolkit.enabled=false

Wait until every pod in the namespace is Running or Completed, then confirm the node advertises GPUs and run NVIDIA's sample workload:

kubectl get pods -n gpu-operator
kubectl get clusterpolicy          # STATUS should be "ready"

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: cuda-vectoradd
spec:
  restartPolicy: OnFailure
  containers:
  - name: cuda-vectoradd
    image: "nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0-ubuntu22.04"
    resources:
      limits:
        nvidia.com/gpu: 1
EOF

kubectl logs pod/cuda-vectoradd    # last lines: "Test PASSED" and "Done"
kubectl delete pod cuda-vectoradd

If the test pod sits in Pending with Insufficient nvidia.com/gpu, the device plugin is not advertising GPUs yet. Look at the validator and device plugin pod logs in gpu-operator before going further; nothing later in this guide will work until this step passes.

Step 3: Create a namespace, model cache PVC and token secret

Model weights are the slow part of every vLLM start. Downloading 16 GB on every pod restart wastes minutes and bandwidth, so mount a persistent volume at the Hugging Face cache path. The example model, Qwen/Qwen3-8B, is Apache 2.0 with a 32,768-token native context and is not gated, so the token is optional; you need it for gated models such as Llama.

# llm-base.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: llm
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: model-cache
  namespace: llm
spec:
  accessModes: ["ReadWriteOnce"]   # use ReadWriteMany (EFS, Filestore, Azure Files) before scaling past 1 replica
  resources:
    requests:
      storage: 100Gi               # room for two or three models of this size
  # storageClassName: <your-class>
---
apiVersion: v1
kind: Secret
metadata:
  name: hf-token
  namespace: llm
type: Opaque
stringData:
  token: "REPLACE_WITH_HF_TOKEN"   # only required for gated models; better: sync from your secret manager

kubectl apply -f llm-base.yaml
kubectl get pvc -n llm   # Bound, or Pending until the first pod mounts it (WaitForFirstConsumer)

Step 4: Deploy vLLM v0.32.0 as a Deployment

This is the core manifest. It is based on the vLLM Kubernetes deployment docs, with three production changes: a pinned image tag instead of latest, a startupProbe so slow model loads are not killed by the liveness probe, and the Recreate strategy so a rollout does not wait forever for a second GPU or try to attach a ReadWriteOnce volume twice.

# vllm-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-qwen3-8b
  namespace: llm
  labels: {app: vllm-qwen3-8b}
spec:
  replicas: 1
  strategy:
    type: Recreate                 # GPU-bound: free the GPU and the RWO volume before starting the new pod
  selector:
    matchLabels: {app: vllm-qwen3-8b}
  template:
    metadata:
      labels: {app: vllm-qwen3-8b}
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: vllm
        image: vllm/vllm-openai:v0.32.0          # CUDA 13.0; use v0.32.0-cu129 on R575 drivers
        command: ["vllm", "serve"]
        args:
        - "Qwen/Qwen3-8B"
        - "--served-model-name=qwen3-8b"        # the name clients send as "model"
        - "--max-model-len=16384"               # below the 32k native max to leave KV cache headroom on 24 GB
        - "--gpu-memory-utilization=0.90"       # fraction of GPU memory vLLM may claim (0.90 is the default)
        - "--max-num-seqs=64"                   # cap concurrent sequences per replica
        - "--reasoning-parser=qwen3"            # split <think> output into reasoning_content
        - "--port=8000"
        env:
        - name: HF_TOKEN
          valueFrom:
            secretKeyRef: {name: hf-token, key: token, optional: true}
        ports:
        - name: http
          containerPort: 8000
        resources:
          requests: {cpu: "4", memory: 24Gi, nvidia.com/gpu: "1"}
          limits:   {cpu: "8", memory: 32Gi, nvidia.com/gpu: "1"}
        volumeMounts:
        - {name: model-cache, mountPath: /root/.cache/huggingface}
        - {name: shm, mountPath: /dev/shm}
        startupProbe:                           # up to 15 minutes for download + load + warmup
          httpGet: {path: /health, port: http}
          periodSeconds: 10
          failureThreshold: 90
        readinessProbe:
          httpGet: {path: /health, port: http}
          periodSeconds: 5
          failureThreshold: 3
        livenessProbe:
          httpGet: {path: /health, port: http}
          periodSeconds: 15
          failureThreshold: 4
      volumes:
      - name: model-cache
        persistentVolumeClaim: {claimName: model-cache}
      - name: shm                               # shared memory for tensor-parallel workers (NCCL)
        emptyDir: {medium: Memory, sizeLimit: 4Gi}
      # Pin to a GPU type if the cluster mixes them (label set by GPU feature discovery):
      # nodeSelector:
      #   nvidia.com/gpu.product: NVIDIA-L4

kubectl apply -f vllm-deployment.yaml
kubectl -n llm rollout status deploy/vllm-qwen3-8b --timeout=20m
kubectl -n llm logs deploy/vllm-qwen3-8b -f

On the first start the log shows the weight download, model loading, the KV cache size vLLM allocated, CUDA graph capture, and finally the API server listening on port 8000. The second start reuses the cache and skips the download.

To serve a bigger model, change the args, not the structure. OpenAI's gpt-oss-120b, for example, fits one 80 GB GPU; the vLLM gpt-oss recipe recommends --max-num-batched-tokens 8192 and tensor parallel size 1 for best per-GPU throughput, and warns that H100 at TP=1 can hit CUDA OOM unless you use --gpu-memory-utilization 0.95 --max-num-batched-tokens 1024. For tensor parallelism across several GPUs on one node, request nvidia.com/gpu: "4" and add --tensor-parallel-size=4.

Step 5: Expose the OpenAI-compatible endpoint and test it

# vllm-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: vllm-qwen3-8b
  namespace: llm
  labels: {app: vllm-qwen3-8b}
spec:
  type: ClusterIP
  selector: {app: vllm-qwen3-8b}
  ports:
  - name: http
    port: 80
    targetPort: http

kubectl apply -f vllm-service.yaml
kubectl -n llm port-forward svc/vllm-qwen3-8b 8000:80 &

curl -s localhost:8000/v1/models | jq '.data[].id'
# "qwen3-8b"

curl -s localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
        "model": "qwen3-8b",
        "messages": [{"role": "user", "content": "Name three Kubernetes workload types."}],
        "max_tokens": 200,
        "chat_template_kwargs": {"enable_thinking": false}
      }' | jq -r '.choices[0].message.content'

Any OpenAI client works by changing the base URL. Qwen3 thinks by default; the vLLM reasoning docs show turning it off per request with chat_template_kwargs, as above, or for every request with --default-chat-template-kwargs.

# pip install openai==2.*   (any recent v1+ client works)
from openai import OpenAI

client = OpenAI(base_url="http://vllm-qwen3-8b.llm.svc.cluster.local/v1", api_key="unused")
resp = client.chat.completions.create(
    model="qwen3-8b",
    messages=[{"role": "user", "content": "One sentence on what a PVC is."}],
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)
print(resp.choices[0].message.content)

vLLM accepts any API key unless you start it with --api-key. Do not expose this Service outside the cluster without authentication in front of it; an AI gateway or your ingress layer is the usual place for keys, rate limits and per-team budgets.

Step 6: Scrape vLLM metrics and autoscale on queue depth with KEDA

GPU utilisation is a poor autoscaling signal for LLMs: a GPU can show high utilisation while serving comfortably, and the thing users feel is requests waiting. vLLM exposes Prometheus metrics on /metrics on the same port, including vllm:num_requests_running, vllm:num_requests_waiting, vllm:kv_cache_usage_perc and the vllm:time_to_first_token_seconds histogram (names from the vLLM metrics design doc). Scale on the waiting queue.

First, get the metrics into Prometheus. With the Prometheus Operator (kube-prometheus-stack), a ServiceMonitor is enough:

# vllm-servicemonitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: vllm
  namespace: llm
  labels: {release: kube-prometheus-stack}   # match your Prometheus serviceMonitorSelector
spec:
  selector:
    matchLabels: {app: vllm-qwen3-8b}
  endpoints:
  - port: http
    path: /metrics
    interval: 15s

Check that the series exists before you build autoscaling on it:

kubectl -n llm exec deploy/vllm-qwen3-8b -- curl -s localhost:8000/metrics | grep -E '^vllm:num_requests_(waiting|running)'
# vllm:num_requests_running{engine="0",model_name="qwen3-8b"} 0.0
# vllm:num_requests_waiting{engine="0",model_name="qwen3-8b"} 0.0

The exact label set can differ between releases, so trust what your pod prints over my sample. Then install KEDA and add a ScaledObject using the Prometheus scaler:

helm repo add kedacore https://kedacore.github.io/charts && helm repo update
helm install keda kedacore/keda -n keda --create-namespace --version 2.21.0

# vllm-scaledobject.yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-qwen3-8b
  namespace: llm
spec:
  scaleTargetRef:
    name: vllm-qwen3-8b
  minReplicaCount: 1          # keep one warm; cold starts take minutes
  maxReplicaCount: 4
  pollingInterval: 15
  cooldownPeriod: 600
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleDown:
          stabilizationWindowSeconds: 600   # do not drop a GPU pod you paid minutes to warm up
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://kube-prometheus-stack-prometheus.monitoring.svc:9090
      query: sum(vllm:num_requests_waiting{model_name="qwen3-8b"})
      threshold: "5"          # target average waiting requests per replica

Two things must change before this scales beyond one pod. Switch the PVC to a ReadWriteMany storage class (or give each pod its own cache), and switch the Deployment strategy back to RollingUpdate with maxSurge: 0, maxUnavailable: 1 if you have no spare GPU for a surge pod. Also make sure your cluster autoscaler or Karpenter can add GPU nodes, otherwise new replicas just sit in Pending.

Step 7: When to move to vLLM production-stack or llm-d

A Deployment plus Service round-robins requests across replicas, which throws away vLLM's prefix cache every time a conversation lands on a different pod. Once you run several replicas or several models, a KV-cache-aware router pays for itself. Two projects package that for Kubernetes:

• vLLM production-stack: a Helm chart (vllm/vllm-stack, latest chart release 0.1.13 on 29 September 2026) that deploys vLLM engines plus a router that routes by session or prefix to reuse KV cache, with Grafana dashboards. Install is helm repo add vllm https://vllm-project.github.io/production-stack then helm install vllm vllm/vllm-stack -f values.yaml.
• llm-d: reached v1.0.0 today, 20 October 2026, with a v1 router configuration API, priority and fairness flow control, router high availability, and a DisaggregatedSet for managing prefill and decode replicas together. It builds on the Kubernetes Gateway API Inference Extension (v1.6.2) and is now steering users to upstream vLLM images rather than its own.

My rule of thumb: one model, one to three replicas, use the manifests above. Many models, long multi-turn sessions, or prefill/decode disaggregation, evaluate llm-d. The inference serving post explains why prefill and decode want different machines.

Troubleshooting

Symptom	Likely cause	Fix
Pod Pending, event Insufficient nvidia.com/gpu	Device plugin not advertising GPUs, or all GPUs taken	Check kubectl get pods -n gpu-operator, the node's allocatable nvidia.com/gpu, and that no other pod holds the GPU
torch.OutOfMemoryError: CUDA out of memory during load or warmup	Model plus KV cache plus CUDA graphs exceed GPU memory	Lower --max-model-len and --max-num-seqs, try --max-num-batched-tokens 1024, or use a quantised model or more GPUs with --tensor-parallel-size
Error that the model's max seq len needs more KV cache than is available	--max-model-len defaults to the model's full context	Set --max-model-len to what you need, or raise --gpu-memory-utilization slightly (0.92 to 0.95) if nothing else shares the GPU
Container restarts during load, log ends with KeyboardInterrupt: terminated	Liveness or startup probe killed a slow start	Use a startupProbe with a large failureThreshold, as in Step 4
First start takes 10+ minutes	Weights downloading from Hugging Face on every start	Mount the PVC at /root/.cache/huggingface; pre-warm it with a Job; keep the image pulled on GPU nodes
CUDA driver version is insufficient or no CUDA-capable device	Image CUDA 13.0 on an older driver	Upgrade to an R580+ driver, or use the -cu129 image tag
NCCL errors or hangs with tensor parallelism	/dev/shm too small (64 MB default)	Mount a memory-backed emptyDir at /dev/shm, as in Step 4
401 or GatedRepoError on download	Gated model and no valid HF_TOKEN, or licence not accepted	Accept the model licence on Hugging Face and fix the secret
Rollout stuck, new pod Pending forever	RollingUpdate needs a second GPU, or RWO volume already attached	Use Recreate, or maxSurge: 0, or a ReadWriteMany volume
KEDA never scales	Query returns nothing (wrong label) or Prometheus not scraping	Run the query in the Prometheus UI; check kubectl describe scaledobject and the ServiceMonitor labels

Fixing CUDA out of memory in vLLM

vLLM allocates almost all of the memory you allow it at start-up: the weights, then CUDA graphs and activation space, then everything left over becomes KV cache. So OOM at start-up is almost always about configuration, not traffic. Work through it in order: set --max-model-len to the context you actually need; reduce --max-num-seqs; reduce --max-num-batched-tokens; if the error is during CUDA graph capture, try --enforce-eager to confirm, then go back to graphs with a smaller capture size. If none of that fits, the model is too big for one GPU: quantise it or add --tensor-parallel-size. Do not raise --gpu-memory-utilization to 0.98 on a GPU that also runs DCGM or another process.

Fixing slow vLLM startup

Startup time is image pull, plus weight download, plus weight load, plus compile and CUDA graph capture. The vLLM image is large, so pre-pull it on GPU nodes (a DaemonSet that runs sleep with the same image works). The PVC removes the download. v0.32.0 also adds vllm download-kernels and vllm preload --preload-autotune to keep FlashInfer kernels and autotune results warm across restarts, per the release notes. Measure each stage from the pod log before optimising, and size your startupProbe to the measured number plus a margin.

FAQ

Do I need the NVIDIA GPU Operator to run vLLM on Kubernetes?

You need the driver, the container toolkit and the device plugin on every GPU node. The GPU Operator is the standard way to manage all of them together, plus DCGM metrics. Some managed Kubernetes offerings install the driver and device plugin themselves; in that case install the operator with driver.enabled=false or follow your provider's GPU docs.

Which vLLM Docker image should I use?

Use vllm/vllm-openai with a pinned version tag, v0.32.0 as of October 2026, and never latest in production. The default tag targets CUDA 13.0; v0.32.0-cu129 targets CUDA 12.9 for older drivers, and ROCm and CPU images are published separately.

How do I autoscale vLLM on Kubernetes?

Scale on queue depth, not CPU or GPU utilisation. Scrape vllm:num_requests_waiting into Prometheus and point a KEDA Prometheus trigger or an HPA external metric at it, keep at least one warm replica, and use a long scale-down window because each new replica takes minutes to load.

Should I use vLLM production-stack or llm-d?

For a single model on a few replicas, plain manifests are enough. production-stack adds a cache-aware router and dashboards with a Helm install. llm-d 1.0 targets larger fleets with Gateway API integration, flow control and prefill/decode disaggregation.

When I put a vLLM deployment in front of real users, the first things I check are boring: that the startup probe matches a measured cold start, that the image tag and driver branch match, that vllm:num_requests_waiting is on a dashboard next to time to first token, and that a second replica exists before anybody calls it production. GPUs make the cluster expensive, not different; the same Kubernetes and SRE discipline applies. Official references: vLLM Kubernetes deployment, NVIDIA GPU Operator getting started, KEDA Prometheus scaler, vLLM production-stack and llm-d.

Related: Kubernetes as the Agent Control Plane in 2026: Agent Sandbox Hit v1.0, 300 Claims a Second, and DRA in Every Supported Release, LLM Inference in 2026: Prefill Is a Different Machine Now, and the KV Cache Has a Disk Tier and GPU Clouds in 2026: H100s at $2.59 an Hour, $104B of CoreWeave Backlog on $35B of Debt, and a Six-Year Bet on Silicon.