How to Deploy an ML Model on Kubernetes with KServe 0.21 (2026)

This guide is for platform and ML engineers who want a model behind an HTTP endpoint on their own Kubernetes cluster without adopting all of Kubeflow. By the end you will have KServe v0.21.0 (released 25 September 2026) running in Standard mode behind Envoy Gateway, a scikit-learn model answering predictions, a Hugging Face LLM on a GPU with the vLLM backend, autoscaling, a canary rollout, and a checklist for the dreaded "InferenceService not ready". Versions used: Kubernetes 1.32 or later (KServe's stated minimum), Helm 3, cert-manager v1.21.2, Envoy Gateway v1.9.2 and KServe v0.21.0. I checked every chart name, value key and field below against the v0.21.0 release artifacts and rendered the charts with helm template; you need cluster-admin on a cluster you can break, and an NVIDIA GPU node only for Step 6.

Step 1: Pick a deployment mode (Standard, Knative or LLMInferenceService)

KServe's admin overview now lists three ways to serve:

• InferenceService, Standard mode (formerly called RawDeployment): plain Kubernetes Deployments, Services, HPA and Gateway API routes. No Knative, no scale-to-zero. This is what KServe recommends for production and for generative workloads.
• InferenceService, Knative mode (formerly Serverless): Knative Serving revisions, scale-to-zero, request-based autoscaling and the classic canaryTrafficPercent rollout.
• LLMInferenceService: a separate CRD for advanced LLM serving (prefix-aware routing, disaggregated prefill and decode) built on llm-d. Worth it at scale, overkill for a first deployment.

Two traps before you start. First, the Helm chart default is still deploymentMode: Knative (I checked values.yaml in the v0.21.0 kserve-resources chart), so if you do not set Standard explicitly, KServe will expect Knative and your InferenceServices will never become ready. Second, many tutorials still say "Serverless" and "RawDeployment"; current docs say Knative and Standard. This guide uses Standard. If you want to understand what Standard mode gives up versus Knative, my Kubernetes autoscaling explainer covers HPA, KEDA and scale-to-zero trade-offs.

Check your cluster first:

kubectl version            # Server Version must be v1.32 or newer
helm version --short       # v3.x
kubectl get nodes -o wide

Step 2: Install cert-manager

KServe's webhooks need TLS certificates, and the docs require cert-manager 1.15.0 or later. KServe's own v0.21.0 install scripts pin v1.17.0; I use the current v1.21.2 release.

helm repo add jetstack https://charts.jetstack.io --force-update

helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager --create-namespace \
  --version v1.21.2 \
  --set crds.enabled=true \
  --wait

kubectl get pods -n cert-manager
# Expect three Running pods: cert-manager, cert-manager-cainjector, cert-manager-webhook

If cert-manager itself misbehaves later, the cert-manager troubleshooting guide covers the usual suspects.

Step 3: Install Envoy Gateway and the KServe Gateway

Standard mode exposes models through Gateway API HTTPRoutes (Kubernetes Ingress with Istio still works, but Gateway API is the recommended path and is where ingress is heading, as covered in migrating from ingress-nginx to Gateway API). The Envoy Gateway quickstart chart also installs the Gateway API CRDs.

helm install eg oci://docker.io/envoyproxy/gateway-helm \
  --version v1.9.2 \
  -n envoy-gateway-system --create-namespace

kubectl wait --timeout=5m -n envoy-gateway-system \
  deployment/envoy-gateway --for=condition=Available

Now create the GatewayClass and the Gateway KServe will attach routes to. The names and label below match what the KServe chart expects by default (kserve/kserve-ingress-gateway):

# gateway.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: envoy
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: kserve-ingress-gateway
  namespace: kserve
spec:
  gatewayClassName: envoy
  infrastructure:
    labels:
      serving.kserve.io/gateway: kserve-ingress-gateway
  listeners:
    - name: http
      protocol: HTTP
      port: 80
      allowedRoutes:
        namespaces:
          from: All

kubectl create namespace kserve
kubectl apply -f gateway.yaml
kubectl get gateway -n kserve
# PROGRAMMED should turn True once Envoy Gateway has created the proxy Service

Add an HTTPS listener with a certificate before anything real goes through this. (Alternatively, setting kserve.controller.gateway.ingressGateway.createGateway=true on the KServe chart renders an identical HTTP-only Gateway for you.)

Step 4: Install KServe v0.21.0 with Helm

In the v0.20 and v0.21 releases there are three charts that matter for InferenceService: CRDs, the controller, and the runtime catalogue. All are published to GHCR; I confirmed the v0.21.0 tags exist for kserve-crd and kserve-resources in the registry.

# 1. CRDs
helm install kserve-crd oci://ghcr.io/kserve/charts/kserve-crd \
  --version v0.21.0 -n kserve

# 2. Controller in Standard mode, routing through the Gateway from Step 3
helm install kserve oci://ghcr.io/kserve/charts/kserve-resources \
  --version v0.21.0 -n kserve \
  --set kserve.controller.deploymentMode=Standard \
  --set kserve.controller.gateway.ingressGateway.enableGatewayApi=true \
  --set kserve.controller.gateway.ingressGateway.kserveGateway=kserve/kserve-ingress-gateway

# 3. ClusterServingRuntimes (sklearn, xgboost, huggingface, vllm, triton, ...)
helm install kserve-runtime-configs oci://ghcr.io/kserve/charts/kserve-runtime-configs \
  --version v0.21.0 -n kserve \
  --set kserve.servingruntime.enabled=true

kubectl get pods -n kserve
kubectl get crd | grep serving.kserve.io
kubectl get clusterservingruntimes

Step 3 of that block is the one people skip. In the v0.21.0 charts the runtime definitions live in kserve-runtime-configs, and kserve.servingruntime.enabled defaults to false. Without it, every InferenceService fails with "No runtime found to support specified framework/version". When I rendered the chart it produced 14 ClusterServingRuntimes, including kserve-sklearnserver, kserve-huggingfaceserver and kserve-vllmserver.

Prefer plain manifests? The release page also ships kserve.yaml and kserve-cluster-resources.yaml, applied with kubectl apply --server-side, followed by patching inferenceservice-config to "defaultDeploymentMode": "Standard" as the install docs show.

Step 5: Deploy a scikit-learn InferenceService

This is the canonical iris example from the KServe getting-started docs. Do not use a namespace labelled control-plane: the webhook skips those, the storage initializer is never injected, and the pod fails looking for /mnt/models.

kubectl create namespace kserve-test

kubectl apply -n kserve-test -f - <<'EOF'
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: sklearn-iris
spec:
  predictor:
    minReplicas: 1
    maxReplicas: 3
    model:
      modelFormat:
        name: sklearn           # matched against ClusterServingRuntime supportedModelFormats
      storageUri: gs://kfserving-examples/models/sklearn/1.0/model
      resources:
        requests: {cpu: 100m, memory: 512Mi}
        limits:   {cpu: "1",  memory: 1Gi}
EOF

kubectl get inferenceservice sklearn-iris -n kserve-test -w
# Wait for READY=True. In Standard mode the URL follows the chart's domainTemplate,
# {{ .Name }}-{{ .Namespace }}.{{ .IngressDomain }}, so expect http://sklearn-iris-kserve-test.example.com

What KServe created for you: a Deployment and a Service named sklearn-iris-predictor, an HPA, and an HTTPRoute on the Gateway. An init container called storage-initializer downloads the model to /mnt/models, then the kserve-container runs the sklearn server.

Test it two ways. Directly against the predictor Service, which isolates the model from the gateway:

cat > iris-input.json <<'EOF'
{"instances": [[6.8, 2.8, 4.8, 1.4], [6.0, 3.4, 4.5, 1.6]]}
EOF

kubectl port-forward -n kserve-test svc/sklearn-iris-predictor 8081:80 &
curl -s -H "Content-Type: application/json" \
  http://localhost:8081/v1/models/sklearn-iris:predict -d @iris-input.json
# {"predictions":[1,1]}

And through the Gateway, using the host name from the InferenceService status:

ENVOY_SVC=$(kubectl get svc -n envoy-gateway-system \
  -l gateway.envoyproxy.io/owning-gateway-name=kserve-ingress-gateway \
  -o jsonpath='{.items[0].metadata.name}')
kubectl port-forward -n envoy-gateway-system svc/${ENVOY_SVC} 8080:80 &

SERVICE_HOSTNAME=$(kubectl get isvc sklearn-iris -n kserve-test \
  -o jsonpath='{.status.url}' | cut -d/ -f3)
curl -s -H "Host: ${SERVICE_HOSTNAME}" -H "Content-Type: application/json" \
  http://localhost:8080/v1/models/sklearn-iris:predict -d @iris-input.json

Both should return two predictions of class 1 (Iris Versicolour). If the first works and the second does not, your model is fine and the problem is routing.

Step 6: Deploy a Hugging Face LLM on a GPU with the vLLM backend

The Hugging Face runtime serves generative models through an OpenAI-compatible API and uses vLLM as its engine where it can. Its runtime arguments include --backend (auto, huggingface or vllm) and --max_model_len, and vLLM engine arguments are passed through. You need the NVIDIA device plugin or GPU Operator so that nvidia.com/gpu is schedulable.

# qwen.yaml
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: qwen-llm
  namespace: kserve-test
spec:
  predictor:
    model:
      modelFormat:
        name: huggingface
      args:
        - --model_name=qwen          # the "model" field clients send
        - --backend=vllm             # force the vLLM engine instead of auto-detecting
        - --max_model_len=8192       # cap KV cache so it fits one small GPU
      storageUri: hf://Qwen/Qwen2.5-0.5B-Instruct
      resources:
        requests: {cpu: "1", memory: 4Gi, nvidia.com/gpu: "1"}
        limits:   {cpu: "2", memory: 6Gi, nvidia.com/gpu: "1"}

kubectl apply -f qwen.yaml
kubectl get isvc qwen-llm -n kserve-test -w

kubectl port-forward -n kserve-test svc/qwen-llm-predictor 8082:80 &
curl -s http://localhost:8082/openai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen","messages":[{"role":"user","content":"Say hello in five words."}],"max_tokens":32}'

Note the /openai/v1 prefix. For gated models, create a secret with HF_TOKEN and a ClusterStorageContainer for the hf:// prefix, as in the KServe LLM quickstart. One thing I would change before production: the kserve-vllmserver runtime in the v0.21.0 chart points at vllm/vllm-openai:latest, an unpinned tag. Pin images in your own ServingRuntime so a rollout cannot change the engine under you. For deeper vLLM tuning on Kubernetes, see deploying vLLM on Kubernetes GPUs.

Step 7: Configure autoscaling with HPA or KEDA

In Standard mode KServe creates an HPA per component. If you do not set replicas, the defaults are minReplicas: 1, maxReplicas: 1, CPU at 80 percent, which means it never actually scales. Set the bounds and the metric explicitly:

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: sklearn-iris
  namespace: kserve-test
  annotations:
    serving.kserve.io/autoscalerClass: hpa
spec:
  predictor:
    minReplicas: 2                 # never 0 in Standard mode; scale-to-zero needs Knative or KEDA
    maxReplicas: 10
    scaleMetric: cpu               # cpu or memory with HPA
    scaleTarget: 70                # percent utilisation
    model:
      modelFormat: {name: sklearn}
      storageUri: gs://kfserving-examples/models/sklearn/1.0/model
      resources:
        requests: {cpu: 100m, memory: 512Mi}
        limits:   {cpu: "1",  memory: 1Gi}

HPA needs metrics-server. CPU is a poor signal for LLMs, where the GPU is the bottleneck and queue depth is what users feel. For that, install KEDA and scale on vLLM metrics from Prometheus, as the KServe LLM autoscaling docs do:

helm repo add kedacore https://kedacore.github.io/charts
helm install keda kedacore/keda --namespace keda --create-namespace --version 2.21.0

metadata:
  annotations:
    serving.kserve.io/autoscalerClass: keda
spec:
  predictor:
    minReplicas: 1
    maxReplicas: 5
    autoScaling:
      metrics:
        - type: External
          external:
            metric:
              backend: prometheus
              serverAddress: http://prometheus.monitoring.svc:9090   # your Prometheus
              query: vllm:num_requests_running
            target:
              type: Value
              value: "2"        # about two in-flight requests per replica

KServe then creates a KEDA ScaledObject named <isvc>-predictor; check it with kubectl get scaledobjects -n kserve-test.

Step 8: Roll out a new model version with a canary

This changed recently, and the docs have not caught up. The website's canary page still says canary rollouts only work in Knative mode, using canaryTrafficPercent. But PR #5672, merged in July 2026 and shipped in v0.20.0, added a spec.canary list for Standard mode, and the v0.21.0 API types define it: each entry has a named predictor and a trafficPercent from 0 to 100, gets its own Deployment called {isvc}-{name}-predictor, and receives traffic through weighted HTTPRoute backends (so Gateway API is required).

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: sklearn-iris
  namespace: kserve-test
spec:
  predictor:
    minReplicas: 4
    model:
      modelFormat: {name: sklearn}
      storageUri: gs://kfserving-examples/models/sklearn/1.0/model
  canary:
    - trafficPercent: 25          # 0 = dark launch, deployed but no traffic
      predictor:
        name: v2                  # required; Deployment becomes sklearn-iris-v2-predictor
        model:
          modelFormat: {name: sklearn}
          storageUri: gs://kfserving-examples/models/sklearn/1.0/model-2

Per the PR, canary replicas are carved out of the stable minReplicas in proportion to traffic (4 replicas at 25 percent gives 1 canary and 3 stable), and canaries may not set their own autoscaling fields. To roll back, delete the canary entry. To promote without restarting pods, set spec.predictor.name: v2 and remove the entry. Because this is new and undocumented on the website, test it in staging and watch kubectl get isvc sklearn-iris -n kserve-test -o yaml for the canary status and conditions. If you need metric-driven analysis and automatic aborts rather than manual weights, a tool like Argo Rollouts is still the more mature option.

For zero-downtime replacement without traffic splitting, set a standard Kubernetes rollout per component; KServe's rollout strategy docs recommend maxUnavailable: "0" for availability:

spec:
  predictor:
    deploymentStrategy:
      type: RollingUpdate
      rollingUpdate:
        maxUnavailable: "0"
        maxSurge: "1"

How to debug "InferenceService not ready"

Work down the stack in this order; each box maps to rows in the table below.

The commands I run, in that order:

NS=kserve-test; ISVC=sklearn-iris
kubectl get isvc $ISVC -n $NS
kubectl get isvc $ISVC -n $NS -o jsonpath='{.status.modelStatus}{"\n"}'
kubectl get isvc $ISVC -n $NS -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason} {.message}{"\n"}{end}'
kubectl get pods -n $NS -l serving.kserve.io/inferenceservice=$ISVC
kubectl describe pod -n $NS -l serving.kserve.io/inferenceservice=$ISVC | tail -30
kubectl logs -n $NS -l serving.kserve.io/inferenceservice=$ISVC -c storage-initializer
kubectl logs -n $NS -l serving.kserve.io/inferenceservice=$ISVC -c kserve-container --tail=100
kubectl logs -n kserve deploy/kserve-controller-manager --tail=100 | grep -i $ISVC
kubectl get httproute -n $NS

Troubleshooting

Symptom	Likely cause	Fix
lastFailureInfo.reason: NoSupportingRuntime, message "No runtime found to support specified framework/version"	kserve-runtime-configs not installed with kserve.servingruntime.enabled=true, or a typo in modelFormat.name	Install the runtime chart (Step 4); kubectl get clusterservingruntimes and match the format name exactly
ISVC never ready, controller logs mention Knative or ksvc	Controller still in default Knative mode	Reinstall kserve-resources with deploymentMode=Standard, or patch inferenceservice-config
Pod stuck Pending	No node with nvidia.com/gpu, quota exceeded, or untolerated taint	kubectl describe pod; install the GPU device plugin, add tolerations, check ResourceQuota. See Pod Pending troubleshooting
Init:Error or Init:CrashLoopBackOff	storage-initializer cannot fetch the model: wrong URI, missing S3/GCS/HF credentials	Read -c storage-initializer logs; fix storageUri, add a credentials secret or ClusterStorageContainer
No such file or directory: '/mnt/models'	ISVC in a namespace labelled control-plane, so the webhook skipped injection	Deploy into a normal namespace
kserve-container OOMKilled (exit 137)	Model or KV cache larger than the memory limit	Raise limits, lower --max_model_len; see OOMKilled exit code 137
ImagePullBackOff on the runtime image	Air-gapped registry or rate-limited latest tag	Mirror and pin the runtime image in your own ServingRuntime; see ImagePullBackOff fixes
Pods Ready, curl to gateway returns 404	Host header does not match the HTTPRoute host from status.url	Use cut -d/ -f3 of status.url as Host, or configure real DNS and gateway.domain
Gateway not PROGRAMMED, curl connection refused	Wrong gatewayClassName, or Envoy Gateway not running	kubectl get gatewayclass, check the envoy-gateway deployment, re-apply Step 3
HPA never scales past 1	Default maxReplicas: 1, or metrics-server missing	Set minReplicas/maxReplicas; install metrics-server; use KEDA for LLM metrics

FAQ

What is the difference between KServe Standard mode and Knative (serverless) mode?

Standard mode creates ordinary Deployments, Services, HPAs and Gateway API routes, so it has fewer moving parts and is recommended for GPUs and LLMs, but it cannot scale to zero with HPA. Knative mode adds revisions, request-based autoscaling and scale-to-zero at the cost of running Knative Serving. The Helm chart default is still Knative, so set Standard explicitly.

Do I need Kubeflow to use KServe?

No. KServe installs on its own with three Helm charts, as in Step 4. Kubeflow bundles an older KServe (0.16.0 in Kubeflow 26.03, 0.18.0 in 26.03.1) alongside Pipelines and notebooks, which is useful only if you want the whole platform.

Why is my InferenceService not ready?

Read status.modelStatus.lastFailureInfo first. The most common causes are a missing runtime (NoSupportingRuntime), a model download failure in the storage-initializer init container, a pod that cannot be scheduled because no GPU is free, or a server that runs out of memory while loading. The flow and table above go through each in order.

Can KServe do canary deployments without Knative?

Yes, since v0.20.0 Standard mode has a spec.canary list with per-canary trafficPercent, split through Gateway API HTTPRoutes. The website docs still describe only the Knative canaryTrafficPercent method, so treat the Standard-mode feature as new and test it before relying on it.

When an InferenceService will not go green, the first thing I check is not the model: it is kubectl get clusterservingruntimes and the controller's deployment mode, because those two account for most of the "it worked in the tutorial" failures I see. In production I pin every runtime image, keep minReplicas at two or more for anything user-facing, scale LLMs on queue depth rather than CPU, and put the GPU endpoints on the same cost dashboard as everything else. If you are still deciding whether to self-host serving at all, my comparison of MLOps platforms in 2026 prices KServe against SageMaker and Vertex. Official references: the KServe documentation, the v0.21.0 release notes, the Gateway API site and the Envoy Gateway docs.

Related: LLM Inference in 2026: Prefill Is a Different Machine Now, Kubernetes as the Agent Control Plane in 2026 and MLOps in 2026: Evals Are the New Unit Tests.