Model Serving Frameworks 2026: Triton vs BentoML vs Ray Serve vs KServe

Every few months someone asks me which model serving framework they should standardise on, and the honest answer has changed more in the last eighteen months than in the five years before. Triton got a new name, BentoML got bought twice in one year, TorchServe stopped being maintained, KServe moved into the CNCF, and for large language models the real decision moved down a layer to engines like vLLM. This post compares the model serving frameworks people actually shortlist in 2026: NVIDIA Dynamo-Triton, BentoML, Ray Serve and KServe, plus where TorchServe and vLLM now sit. I cover what each one is, how it batches, how it handles many models, how it autoscales, and which one I would pick for which job.

What is a model serving framework, and why are there so many?

A model server turns a trained artefact (an ONNX file, a PyTorch checkpoint, a folder of safetensors) into a network endpoint that handles concurrency, batching, health checks and metrics. That sounds like one job. In practice it is three, and most of the confusion in comparisons comes from mixing them up.

The first layer is the inference engine or model server that runs tensors on hardware: Dynamo-Triton, vLLM, SGLang, TensorRT-LLM. The second is the serving framework where you write Python around the model: preprocessing, routing, chaining models, business logic. Ray Serve and BentoML live here. The third is the platform layer that makes it a Kubernetes resource with rollouts and scale-to-zero: KServe, and KubeRay for Ray. A sensible stack often uses one from each layer. KServe can run Triton or vLLM as its runtime; Ray Serve can wrap vLLM; Triton has a vLLM backend.

So "Triton vs Ray Serve" is often the wrong question. The useful question is which layer you are missing.

The 2026 model serving frameworks at a glance

All versions below come from each project's GitHub releases page, checked today.

Project	Latest release	Layer	Batching	Multi-model	Autoscaling	Status and owner	Source
Dynamo-Triton	2.74.0, 2 Nov 2026 (NGC 26.10)	Engine/server	Dynamic batcher in server config	Many models per server, ensembles	External (HPA, KServe)	NVIDIA, BSD-3	releases
BentoML	1.4.40, 23 Oct 2026	Framework	Adaptive batching per API	Multiple services composed in one project	BentoCloud, or your own HPA	Modular (Feb 2026), now Qualcomm; Apache-2.0	releases
Ray Serve	Ray 2.60.0, 4 Nov 2026	Framework	@serve.batch	Composition, model multiplexing	Replica autoscaler plus Ray node autoscaler	PyTorch Foundation since Oct 2025	releases
KServe	0.21.0, 25 Sep 2026	Platform	Optional batcher sidecar	ModelMesh, multiple runtimes	Knative, HPA or KEDA	CNCF incubating since Sep 2025	releases
vLLM	0.32.0, 18 Oct 2026	LLM engine	Continuous batching	One model per server (plus LoRA adapters)	External	PyTorch Foundation	releases
TorchServe	0.12.0, 30 Sep 2024	Server	Server-side batching	Yes	External	Archived, limited maintenance	repo

NVIDIA Dynamo-Triton: the fastest path from tensor to endpoint

Triton Inference Server was renamed NVIDIA Dynamo-Triton in March 2025 when NVIDIA folded it into the Dynamo platform; the product page now calls it "formerly NVIDIA Triton Inference Server". The binary, the repository and the tritonserver command did not change, and release 2.74.0 shipped on 2 November 2026 as the 26.10 NGC container. Its vLLM backend now supports vLLM 0.30 and later.

Triton's strength is that it serves many frameworks from one process: TensorRT, ONNX Runtime, PyTorch, OpenVINO, Python and RAPIDS FIL, all loaded from a model repository directory. Each model gets a config.pbtxt, and batching is declared there rather than written in code:

# models/resnet50/config.pbtxt  (served by nvcr.io/nvidia/tritonserver:26.10-py3)
name: "resnet50"
platform: "onnxruntime_onnx"
max_batch_size: 32                  # largest batch the model accepts
dynamic_batching {
  preferred_batch_size: [ 8, 16 ]   # batch sizes the scheduler aims for
  max_queue_delay_microseconds: 2000  # wait at most 2 ms to fill a batch
}
instance_group [
  { count: 2, kind: KIND_GPU }      # two copies of the model per GPU
]

The dynamic batcher docs explain the trade: requests queue for up to max_queue_delay_microseconds so the server can form bigger batches, which raises GPU throughput at the cost of a little latency. Ensembles let you chain a tokenizer, a model and a postprocessor inside the server without a network hop.

What Triton does not do is scale itself, deploy itself, or make writing Python business logic pleasant. The Python backend exists, but it is a model-shaped hole for code, not a web framework. That is why Triton most often appears underneath KServe or behind a thin gateway. If your models are already ONNX or TensorRT engines and your team thinks in GPU utilisation, Triton is still the most efficient general-purpose server. The release notes are worth reading closely, though: 2.74.0 lists Ray and vLLM's v0 API as affected by vulnerabilities and tells users not to expose Ray executors or vLLM hosts to untrusted networks.

BentoML: Python-first, and now owned by a chip company

BentoML is the framework that feels most like writing a FastAPI app. You decorate a class with @bentoml.service, decorate methods with @bentoml.api, and bentoml build packages code, model and dependencies into a "bento" that becomes an OCI image. Adaptive batching is a flag on the API: @bentoml.api(batchable=True), with max_batch_size defaulting to 100 and max_latency_ms to 60,000 in the 1.4.40 source. The adaptive batching guide covers tuning.

Ownership is the part that changed. Modular announced on 10 February 2026 that BentoML was joining it, with the open-source project continuing under Apache-2.0 with the same docs and contribution process. Then Qualcomm announced it would buy Modular in June and completed the acquisition on 29 July 2026. Releases have kept coming; 1.4.40 on 23 October was a security release that requires certifi>=2026.10.19 after the eMudhra CA compromise and tells users to rebuild and redeploy their bentos.

I do not think the acquisition makes BentoML risky to use today. It does mean the commercial roadmap (BentoCloud, its autoscaler, its external_queue option, which the source marks as BentoCloud only) belongs to a company whose interests are hardware. If you self-host on Kubernetes, you supply autoscaling yourself with an HPA or KEDA, as you would for any container.

Ray Serve: when the model is one step in a Python program

Ray Serve is a library on top of Ray, so a deployment is a Ray actor and composition is just one deployment holding a handle to another. That makes it strong for pipelines: an ingress that calls an embedding model, a reranker and an LLM, each scaled independently, each with its own CPU or GPU request. It also supports model multiplexing, where many fine-tuned variants share a pool of replicas and requests carry a model ID.

Batching is a decorator, @serve.batch, which defaults to max_batch_size=10 and batch_wait_timeout_s=0.01 in Ray 2.60.0 (batching guide). Autoscaling has two halves. Serve adds or removes replicas to keep each near target_ongoing_requests (default 2), and the Ray autoscaler adds or removes nodes, which on Kubernetes means KubeRay adds or removes Pods (autoscaling guide). For LLMs there is ray.serve.llm, which wraps vLLM behind an OpenAI-compatible app (Serve LLM docs).

Ray 2.60.0, released on 4 November, added max_surge_percent for rolling updates that do not drop capacity, and failed rolling updates now stop at DEPLOY_FAILED while old replicas keep serving (release notes). Ray's token authentication docs warn that Ray 2.61 turns on token authentication for every cluster by default, so every node and client will need the same token. Plan for that before you upgrade. Governance moved too: Ray became a PyTorch Foundation project in October 2025, next to PyTorch and vLLM.

The cost of Ray Serve is Ray itself. You are running a distributed system with a head node, a control store and its own scheduler inside Kubernetes's scheduler. KubeRay's RayService makes that manageable, and I wrote up the full setup in How to Deploy Ray Serve on Kubernetes with KubeRay, but it is more moving parts than a Deployment and an HPA.

KServe: the Kubernetes-native control plane

KServe is not a model server in the Triton sense. It is a set of CRDs and controllers that turn "serve this model from this bucket with this runtime" into Pods, routes, canaries and autoscaling. The InferenceService resource picks a serving runtime (its own Python servers, Triton, vLLM and others) and adds an optional batcher sidecar configured with maxBatchSize and maxLatency (batcher docs). For LLMs, the newer LLMInferenceService integrates with llm-d style routing; the 0.21.0 notes are full of llmisvc work, canary tests, DRA resourceClaims support in serving runtimes, and samples moving from v1alpha1 to v1alpha2.

KServe was accepted as a CNCF incubating project in September 2025, which matters if you care about neutral governance. Its weakness is the dependency footprint: serverless mode wants Knative and a networking layer, and even the standard (raw deployment) install pulls in cert-manager and, for most setups, a Gateway API implementation. I have a step-by-step in How to Deploy an ML Model on Kubernetes with KServe 0.21.

Is TorchServe deprecated? What to use instead

Effectively, yes. The TorchServe docs carry a "Limited Maintenance" notice saying there are no planned updates, bug fixes, new features or security patches, and the GitHub repository is archived and read-only. The last release is 0.12.0 from 30 September 2024. Existing deployments keep working, but every new CVE in its dependency tree is now yours.

Migration targets depend on why you chose it. If you liked handler files and a model archive, KServe's PyTorch runtime or Triton's PyTorch backend are the closest shape. If you mostly wrote Python handlers with pre- and post-processing, BentoML or Ray Serve will feel more natural. If you were serving LLMs through TorchServe, move to vLLM directly.

Which model serving framework should you use for LLMs?

For LLMs, the engine matters more than the framework. Continuous batching, paged KV cache, prefix caching and speculative decoding live in vLLM, SGLang and TensorRT-LLM, and none of the frameworks above reimplement them. They wrap them. My comparison of the engines is in LLM Inference Engines 2026, and the plain Kubernetes path is in How to Deploy vLLM on Kubernetes with GPUs.

The framework question for LLMs becomes: what do you need around the engine? For a single model behind an OpenAI-compatible API, a vLLM Deployment plus an autoscaler is enough. If you need multi-node prefill/decode disaggregation and KV-aware routing, that is what NVIDIA Dynamo (1.6.2, 5 November 2026) and KServe's LLMInferenceService target. If the LLM is one step in a pipeline with retrieval and reranking, Ray Serve LLM or BentoML give you the glue. The platform discipline still matters more than the choice. Pin images, set resource requests that match the GPU, alert on queue depth rather than CPU, and rehearse an upgrade before you need one.

FAQ

Is Triton Inference Server still free and open source?

Yes. The renamed NVIDIA Dynamo-Triton is still developed in the open on GitHub under a BSD-3-Clause licence, and release 2.74.0 shipped on 2 November 2026. NVIDIA sells enterprise support through NVIDIA AI Enterprise, but the server and containers remain free to use.

Ray Serve vs KServe: which is better on Kubernetes?

They solve different problems. KServe is a Kubernetes control plane for single-model endpoints with canaries and scale-to-zero, and it can run Triton or vLLM. Ray Serve is a Python framework for composing several models and business logic, deployed on Kubernetes through KubeRay's RayService. Pick KServe for many independent models; pick Ray Serve for multi-step pipelines.

What replaced TorchServe?

There is no official successor. TorchServe's docs say it is in limited maintenance and the repository is archived. Most teams move to KServe or Dynamo-Triton for handler-style serving, to BentoML or Ray Serve for Python-heavy services, and to vLLM or SGLang for LLMs.

Do I need a serving framework for one LLM?

Usually not. vllm serve exposes an OpenAI-compatible API with continuous batching built in. Add a framework when you need multiple models, custom pre- and post-processing, or independent scaling of pipeline stages.

What I would do this quarter

If I were picking today, I would start from the layer I was missing. For a team with a handful of LLMs I would run vLLM directly and spend the effort on autoscaling and observability. For classical models in ONNX or TensorRT, I would run Triton under KServe. For pipelines where a request touches three models and some Python, I would use Ray Serve on KubeRay, and I would set up Ray token authentication now rather than discover it during the 2.61 upgrade. Anything still on TorchServe goes on the migration list with a date next to it. I build PandaStack, a Firecracker microVM cloud for agents and code execution, so I spend more time on the isolation side than on model serving, but the rule is the same there: choose the component whose failure modes you can operate, not the one with the best benchmark.

Related: MLOps Platforms Compared 2026: MLflow vs Kubeflow vs SageMaker vs Vertex, GPUs on Kubernetes Explained (2026): Device Plugin, MIG, Time-Slicing, DRA and Self-Hosting LLMs vs API Costs in 2026: The 62% Utilisation Break-Even.