How to Deploy Ray Serve on Kubernetes with KubeRay 1.7 (2026)

This guide is for platform and ML engineers who need a Ray Serve application running on Kubernetes the way the KubeRay maintainers intend: as a RayService custom resource that the operator reconciles, upgrades and heals. At the end you will have the KubeRay operator installed with Helm, a Python Serve app with request batching, a RayService with replica and node autoscaling, an optional GPU worker group, and a tested upgrade path. Everything is pinned to KubeRay 1.7.2 (released 6 November 2026) and Ray 2.60.0 (4 November 2026). I ran the Serve app locally on Ray 2.60.0 and validated every manifest against the 1.7.2 CRD schema; I did not run the cluster steps for this post, so expected output comes from the official docs. You need a Kubernetes cluster you can install CRDs into, Helm 3.x or 4.x, kubectl, and a container registry.

If you are still deciding whether Ray Serve is the right tool at all, read the companion piece first: Model Serving Frameworks 2026: Triton vs BentoML vs Ray Serve vs KServe.

Step 1: Check versions and prerequisites

KubeRay and Ray version independently. The operator image tag and the Helm chart version match (1.7.2), while rayVersion in your manifest must match the Ray version baked into your container image. Mismatch them and the autoscaler sidecar, which KubeRay picks based on rayVersion, will not match the head.

# Tools on your workstation
kubectl version --client
helm version --short          # v3.x or v4.x both work for this chart

# Latest releases at the time of writing
#   KubeRay operator / chart: 1.7.2  (6 Nov 2026)
#   Ray:                      2.60.0 (4 Nov 2026)
#   Image: rayproject/ray:2.60.0-py312 (CPU), rayproject/ray:2.60.0-py312-gpu (CUDA)

The KubeRay repository moved to the kuberay-project GitHub organisation (the old ray-project/kuberay URLs redirect), but the Helm repository is still served at ray-project.github.io/kuberay-helm. I checked that index lists 1.7.2.

Step 2: Install the KubeRay operator with Helm

The current docs recommend a dedicated ray-system namespace rather than default, so the operator's service account is isolated from workload Pods.

helm repo add kuberay https://ray-project.github.io/kuberay-helm/
helm repo update

kubectl create namespace ray-system
helm install kuberay-operator kuberay/kuberay-operator \
  --version 1.7.2 -n ray-system

kubectl get pods -n ray-system
# NAME                                READY   STATUS    RESTARTS   AGE
# kuberay-operator-6bc45dd644-gwtqv   1/1     Running   0          24s

kubectl get crd | grep ray.io
# rayclusters.ray.io, raycronjobs.ray.io, rayjobs.ray.io, rayservices.ray.io

The chart installs the CRDs and the operator together. If your CRDs are managed by a cluster admin, install them separately and pass --skip-crds. In 1.7 the RayServiceIncrementalUpgrade feature gate is beta and enabled by default in the chart's featureGates values, which you will use in Step 8.

Step 3: Write and test a Ray Serve application locally

This app has two deployments: an Ingress that parses HTTP and a Scorer that does the work with @serve.batch. The scorer is a stand-in; replace its body with your model.

# sentiment.py: a Ray Serve app with an ingress and a batched model deployment.
from typing import List

from ray import serve
from ray.serve.handle import DeploymentHandle
from starlette.requests import Request


@serve.deployment(
    ray_actor_options={"num_cpus": 1},
    max_ongoing_requests=16,  # per-replica concurrency cap (default is 5)
    autoscaling_config={
        "min_replicas": 1,
        "max_replicas": 4,
        "target_ongoing_requests": 8,  # default is 2
        "upscale_delay_s": 10,
        "downscale_delay_s": 120,
    },
)
class Scorer:
    def __init__(self, positive_words: List[str]):
        # Stand-in for a real model load (e.g. a transformers pipeline).
        self.positive = set(positive_words)
        self.threshold = 1

    def reconfigure(self, config: dict):
        # Called when user_config is set or changes: no replica restart needed.
        self.threshold = config.get("threshold", 1)

    @serve.batch(max_batch_size=8, batch_wait_timeout_s=0.02)
    async def score(self, texts: List[str]) -> List[dict]:
        # Receives up to 8 queued requests as one list; returns one result each.
        out = []
        for t in texts:
            hits = sum(w in self.positive for w in t.lower().split())
            out.append({"text": t, "positive": hits >= self.threshold, "batch": len(texts)})
        return out


@serve.deployment(num_replicas=1, ray_actor_options={"num_cpus": 0.5})
class Ingress:
    def __init__(self, scorer: DeploymentHandle):
        self.scorer = scorer

    async def __call__(self, request: Request) -> dict:
        body = await request.json()
        return await self.scorer.score.remote(body["text"])


app = Ingress.bind(Scorer.bind(["good", "great", "fast", "reliable"]))

The @serve.batch defaults in Ray 2.60.0 are max_batch_size=10 and batch_wait_timeout_s=0.01; I set them explicitly so the trade is visible. Note that max_ongoing_requests must be at least max_batch_size times max_concurrent_batches, or Serve warns that batches can never fill.

Run it locally before going anywhere near a cluster:

python -m venv .venv && . .venv/bin/activate
pip install "ray[serve]==2.60.0"
serve run sentiment:app

# second terminal: 12 concurrent requests
for i in $(seq 1 12); do
  curl -s -X POST localhost:8000/ -H 'Content-Type: application/json' \
    -d "{\"text\":\"req $i good\"}" &
done; wait

On my machine the concurrent requests came back with "batch":5 and "batch":2, so batching works. Ray 2.60.0 also logged Token authentication is enabled for this Ray cluster and wrote a token to ~/.ray/auth_token; since 2.59 that is the default for local clusters. The Serve endpoint on port 8000 still answered without a token. Then generate the config Serve expects:

serve build sentiment:app -o serve_config.yaml
# The file starts: "# This file was generated using the `serve build` command on Ray v2.60.0."

Step 4: Build a container image with your code

You can point runtime_env.working_dir at a zip URL, which is what the upstream samples do, but in production I bake code into the image so a Pod restart never depends on a download.

# Dockerfile
FROM rayproject/ray:2.60.0-py312
WORKDIR /home/ray/app
COPY sentiment.py .
# Make sentiment:app importable from every Ray worker process
ENV PYTHONPATH=/home/ray/app

docker build -t registry.example.com/ml/sentiment:0.1.0 .
docker push registry.example.com/ml/sentiment:0.1.0

Use the same image for head and workers. A head without your dependencies is the most common reason Serve apps fail to import.

Step 5: Define the RayService

serveConfigV2 is the multi-application Serve config as a YAML string (from Step 3, trimmed). rayClusterConfig is a normal RayCluster spec.

# rayservice.yaml  (KubeRay 1.7.2, Ray 2.60.0)
apiVersion: ray.io/v1
kind: RayService
metadata:
  name: sentiment
  namespace: ml
  annotations:
    ray.io/initializing-timeout: "30m"   # fail instead of hanging in Initializing
spec:
  deploymentUnhealthySecondThreshold: 300
  upgradeStrategy:
    type: NewCluster                     # blue/green on rayClusterConfig changes
  serveConfigV2: |
    applications:
      - name: sentiment
        import_path: sentiment:app
        route_prefix: /
        deployments:
          - name: Scorer
            max_ongoing_requests: 16
            user_config:
              threshold: 1
            autoscaling_config:
              min_replicas: 1
              max_replicas: 4
              target_ongoing_requests: 8
            ray_actor_options:
              num_cpus: 1
          - name: Ingress
            num_replicas: 1
            ray_actor_options:
              num_cpus: 0.5
  rayClusterConfig:
    rayVersion: "2.60.0"                 # must match the image
    enableInTreeAutoscaling: true        # adds the autoscaler sidecar to the head
    autoscalerOptions:
      version: v2
      upscalingMode: Default
      idleTimeoutSeconds: 120
    headGroupSpec:
      rayStartParams:
        num-cpus: "0"                    # keep Serve replicas off the head Pod
      template:
        spec:
          containers:
            - name: ray-head
              image: registry.example.com/ml/sentiment:0.1.0
              resources:
                requests: {cpu: "1", memory: 4Gi}
                limits: {cpu: "2", memory: 4Gi}
              ports:
                - {containerPort: 6379, name: gcs-server}
                - {containerPort: 8265, name: dashboard}
                - {containerPort: 10001, name: client}
                - {containerPort: 8000, name: serve}
    workerGroupSpecs:
      - groupName: cpu
        replicas: 1
        minReplicas: 1
        maxReplicas: 4
        rayStartParams: {}
        template:
          spec:
            containers:
              - name: ray-worker
                image: registry.example.com/ml/sentiment:0.1.0
                lifecycle:
                  preStop:
                    exec:
                      command: ["/bin/sh", "-c", "ray stop"]
                resources:
                  requests: {cpu: "2", memory: 4Gi}
                  limits: {cpu: "2", memory: 4Gi}

Step 6: Apply it and send traffic

kubectl create namespace ml
kubectl apply -f rayservice.yaml

kubectl get rayservice -n ml
# NAME        SERVICE STATUS   NUM SERVE ENDPOINTS
# sentiment   Running          1

kubectl get pods -n ml -l ray.io/is-ray-node=yes
kubectl describe rayservice sentiment -n ml | grep -A6 Conditions
#   Reason:  NonZeroServeEndpoints
#   Status:  True
#   Type:    Ready

kubectl get svc -n ml
# sentiment-head-svc   ...  10001/TCP,8265/TCP,6379/TCP,8080/TCP,8000/TCP
# sentiment-serve-svc  ...  8000/TCP

kubectl run curl -n ml --rm -it --image=curlimages/curl:8.17.0 -- \
  curl -s -X POST sentiment-serve-svc:8000/ \
  -H 'Content-Type: application/json' -d '{"text":"great and fast"}'

Point your Ingress or HTTPRoute at sentiment-serve-svc, never at a generated sentiment-raycluster-xxxxx service. KubeRay repoints the stable services to the new cluster during upgrades; the generated names change.

For the dashboard, kubectl port-forward -n ml svc/sentiment-head-svc 8265:8265 and open http://localhost:8265/#/serve. If you enable token auth (Step 9), the dashboard prompts for the token.

Step 7: Tune autoscaling (replicas and Pods)

There are two autoscalers and they must agree. Serve scales replicas to keep each near target_ongoing_requests; the Ray autoscaler turns unmet resource demand into more worker Pods up to maxReplicas; idle Pods go after idleTimeoutSeconds. Defaults in Ray 2.60.0 are upscale_delay_s: 30 and downscale_delay_s: 600.

Do the arithmetic once: four Scorer replicas at one CPU each plus the ingress need 4.5 CPUs, and each worker offers two, so maxReplicas: 4 on the worker group is enough. If Serve's maximum needs more than the worker group can ever provide, the autoscaler logs No available node types can fulfill resource requests and replicas stay pending.

Edit autoscaling_config in serveConfigV2 and reapply. That is an in-place update: no new RayCluster. Editing replicas, minReplicas, maxReplicas or scaleStrategy.workersToDelete under rayClusterConfig is deliberately ignored, because the autoscaler owns those fields. Your cluster autoscaler or Karpenter must also be able to add nodes, otherwise new Pods sit in Pending; see Kubernetes Autoscaling Explained.

Step 8: Add a GPU worker group

Install the NVIDIA device plugin or GPU Operator first (guide). Build a second image FROM rayproject/ray:2.60.0-py312-gpu with your ML libraries, then add a scale-from-zero group:

# append under spec.rayClusterConfig.workerGroupSpecs
      - groupName: gpu
        replicas: 0
        minReplicas: 0
        maxReplicas: 2
        rayStartParams: {}
        template:
          spec:
            tolerations:
              - key: nvidia.com/gpu
                operator: Exists
                effect: NoSchedule
            containers:
              - name: ray-worker
                image: registry.example.com/ml/sentiment-gpu:0.1.0
                resources:
                  requests: {cpu: "4", memory: 16Gi, nvidia.com/gpu: "1"}
                  limits: {cpu: "4", memory: 16Gi, nvidia.com/gpu: "1"}

# and in serveConfigV2, ask for a GPU per replica
          - name: Scorer
            ray_actor_options:
              num_cpus: 1
              num_gpus: 1

KubeRay reads the nvidia.com/gpu limit and starts Ray with the matching --num-gpus, so the autoscaler knows that group supplies GPUs. One gotcha from the KubeRay GPU docs: a Pod without an nvidia.com/gpu request on a GPU node sees all GPUs because NVIDIA_VISIBLE_DEVICES defaults to all; set it to void on CPU Pods that can land there.

Step 9: Upgrade without downtime

App changes (user_config, deployment options, autoscaling, a runtime_env change) go in serveConfigV2 and are applied in place. user_config changes call reconfigure() on live replicas without restarting them.

Cluster changes (image, resources, rayVersion) trigger the NewCluster strategy: KubeRay creates a second RayCluster, waits for its Serve apps to be healthy, switches the stable services, and deletes the old one. That needs 200 percent of capacity for a while, which is painful for GPUs. KubeRay 1.7 promoted NewClusterWithIncrementalUpgrade to beta. It shifts traffic in steps through Gateway API, so it needs Gateway API CRDs, a Gateway controller and a GatewayClass, plus in-tree autoscaling:

spec:
  upgradeStrategy:
    type: NewClusterWithIncrementalUpgrade
    clusterUpgradeOptions:
      gatewayClassName: istio      # required: your GatewayClass
      maxSurgePercent: 25          # extra capacity per step (default 100)
      stepSizePercent: 5           # required: traffic moved per interval
      intervalSeconds: 30          # required: pace of the shift

kubectl get httproute -n ml sentiment-httproute -o yaml | grep -B2 weight
kubectl describe rayservice sentiment -n ml | grep -E 'Target Capacity|Traffic Routed Percent'

If you are new to Gateway API, How to Migrate From ingress-nginx to Gateway API covers the controller side. Before the Ray 2.61 upgrade, which Ray's token authentication docs say will make token authentication the default for all clusters, add authOptions: {mode: token} under rayClusterConfig. KubeRay 1.6 and later then create a Secret with a random token and inject RAY_AUTH_MODE and RAY_AUTH_TOKEN into every Ray container.

Troubleshooting

Symptom	Likely cause	Fix
RayService stays Initializing forever	Pods in ImagePullBackOff or CrashLoopBackOff	Fix the image; add ray.io/initializing-timeout: "30m" so it fails visibly
App status DEPLOY_FAILED, ModuleNotFoundError	Wrong import_path or code not on PYTHONPATH in the image	kubectl exec into the head and python -c "import sentiment"
Operator log: connect: connection refused to the head	Dashboard not up yet, or crashed	Wait a minute; check /tmp/ray/session_latest/logs/dashboard.log on the head
Replicas pending, No available node types can fulfill resource requests	Serve asks for CPU/GPU no worker group provides	Raise maxReplicas, add a GPU group, or lower num_cpus/num_gpus
Worker Pods Pending	Kubernetes has no node with capacity	Check cluster autoscaler or Karpenter, taints and tolerations
Changing worker replicas does nothing	Autoscaler-managed fields are not propagated	Let the autoscaler own Pod counts; tune Serve autoscaling_config instead
Upgrade hangs with two RayClusters	NewCluster needs 2x resources	Add capacity, or switch to NewClusterWithIncrementalUpgrade
Incremental upgrade never shifts traffic	Missing Gateway API CRDs, GatewayClass, or autoscaling	Install CRDs and a controller, set gatewayClassName, enable in-tree autoscaling
Autoscaler sidecar errors after image bump	rayVersion does not match the image	Set rayVersion to the Ray version in the image
Authentication required: Unauthorized from Ray CLI	Token auth on, client has no token	Export RAY_AUTH_MODE=token and RAY_AUTH_TOKEN from the Secret

Start every investigation with kubectl describe rayservice, the operator logs (kubectl logs -n ray-system deploy/kuberay-operator) and the head Pod logs, in that order.

FAQ

What is the difference between RayService and RayCluster?

A RayCluster is just Ray: a head and worker groups. A RayService owns a RayCluster plus the Serve config, adds health-aware Services, applies Serve changes in place, and replaces the cluster safely when its spec changes. For long-running inference use RayService; for batch jobs use RayJob.

Does Ray Serve on KubeRay scale to zero?

Worker groups can go to zero (minReplicas: 0) and Serve deployments can use min_replicas: 0, but the head Pod always runs. Expect a cold start of Pod scheduling plus model load when traffic returns, which for GPU nodes can be minutes.

Can I use Helm 4 with the KubeRay chart?

The chart README still documents Helm 3, and I tested rendering with Helm 3.14. The install commands are the same under Helm 4; check the Helm 4 changes if you rely on post-renderers.

Should I serve LLMs with Ray Serve or plain vLLM?

For a single model, a plain vLLM Deployment is simpler. Use ray.serve.llm when you need several models, multi-step pipelines or Ray's autoscaling across heterogeneous GPU groups.

What I check first in production

When a RayService misbehaves I look at three things before anything else: whether rayVersion matches the image, whether Serve's maximum resource demand fits inside the worker groups' maxReplicas, and whether traffic goes through the stable -serve-svc. Between them they explain most rows in the troubleshooting table above. I also keep app config changes and cluster changes in separate pull requests, so I always know whether an apply will be an in-place update or a full blue/green. The official references are the KubeRay RayService guide, the incremental upgrade guide, KubeRay autoscaling, RayService troubleshooting, KubeRay token authentication and the KubeRay 1.7.0 release notes.

Related: How to Deploy vLLM on Kubernetes with GPUs: Step-by-Step (2026), How to Deploy an ML Model on Kubernetes with KServe 0.21 (2026) and GPUs on Kubernetes Explained (2026): Device Plugin, MIG, Time-Slicing, DRA.