LLM Inference Engines 2026: vLLM vs SGLang vs TensorRT-LLM vs Dynamo

Every few weeks someone asks me which LLM inference engine they should run, and they usually want a one-word answer: vLLM, SGLang, or TensorRT-LLM. I build infrastructure for AI agents rather than for model serving, but the agents spend most of their time waiting on an inference endpoint, so I keep reading the release notes and benchmark repos. The honest answer for 2026 is that the engines have converged on the same core ideas, the published benchmark gaps are smaller than the gaps between a tuned and an untuned config, and the choice is mostly about hardware, model support and who operates it. This post compares the main LLM inference engines as of 31 October 2026, explains what the shared techniques do, looks at what the published numbers actually show, and ends with which one I would pick for which job.

If you want the architecture story (prefill/decode split, KV cache tiers, wide expert parallelism), I covered that in LLM Inference in 2026: Prefill Is a Different Machine Now. This one is the buyer's guide.

Which LLM inference engines matter in 2026?

Five open-source projects cover nearly every self-hosted deployment I see, plus one orchestration layer that sits above them. Here is where each stands today, with versions taken from the GitHub releases pages.

Engine	Latest release (31 Oct 2026)	Best at	Status / source
vLLM	v0.32.0, 18 Oct 2026	Broadest model and hardware support, OpenAI-compatible server	vLLM releases
SGLang	v0.5.22, 19 Oct 2026	Prefix-heavy and structured workloads, large MoE at scale	SGLang releases
TensorRT-LLM	v1.2.1 stable (20 Apr 2026); v1.4.0rc2 on 28 Oct	Peak per-GPU numbers on NVIDIA, especially Blackwell	TensorRT-LLM releases
NVIDIA Dynamo	v1.6.1, 27 Oct 2026	Multi-node orchestration over vLLM, SGLang or TensorRT-LLM	Dynamo releases
llama.cpp	v0.8.0, 28 Oct 2026 (daily builds up to b12019)	CPUs, Apple Silicon, consumer GPUs, GGUF quantisation	llama.cpp releases
Hugging Face TGI	v3.3.7, 19 Dec 2025	Nothing new: maintenance mode, repository archived	TGI README

Two things in that table are worth a sentence each. TensorRT-LLM ships release candidates roughly weekly, but the last non-RC tag is still 1.2.1 from April, so "latest TensorRT-LLM" means very different things to different people. And llama.cpp, which for years had only monotonically increasing build numbers, now also cuts semantic-version releases: v0.1.0 was tagged in August 2026 and v0.8.0 on 28 October, while bNNNNN builds still land several times a day. Pin a release tag, not a build number, if you want something you can reason about.

Is TGI deprecated? What happened to Text Generation Inference

Effectively, yes. The TGI README now opens with a caution that the project is in maintenance mode and will only accept minor bug fixes, documentation and lightweight maintenance. It recommends vLLM and SGLang for servers, and llama.cpp or MLX for local inference. The GitHub repository is now archived (read-only), and the last release, v3.3.7, shipped on 19 December 2025.

Hugging Face's argument is that TGI's real contribution was getting engines to reuse transformers model definitions, and that vLLM and SGLang now do that, so a third engine is redundant. Whatever you think of that, the operational conclusion is simple: if you run TGI in production, plan the migration this quarter. Both vLLM and SGLang expose OpenAI-compatible endpoints, so the client side is usually a URL change. The work is in re-validating latency and output quality, which is exactly what the sibling guide, How to Benchmark LLM Inference, walks through.

The other ownership change this year went the opposite direction. The ggml.ai team behind llama.cpp joined Hugging Face in February 2026, with the projects staying open source. So Hugging Face now backs a local engine rather than a server engine.

How do continuous batching, PagedAttention and RadixAttention work?

All the server engines share three ideas, and knowing them makes the benchmark claims readable.

Continuous batching. A GPU decoding one request at a time is mostly idle, because each token step is bound by reading the weights from memory, not by arithmetic. Batching many requests lets one weight read serve many tokens. Continuous (or in-flight) batching admits new requests and retires finished ones at every decode step instead of waiting for a whole batch to finish. Every engine in the table does this, including llama-server with its parallel slots.

PagedAttention. The KV cache, the per-request attention state, grows token by token and its final size is unknown. Allocating it contiguously wastes memory on fragmentation and over-reservation, which caps batch size. The PagedAttention paper borrowed virtual memory paging: the cache is split into fixed-size blocks mapped through a block table. The paper reported near-zero KV waste and 2 to 4 times the throughput of FasterTransformer and Orca at the same latency. Paged KV caches are now standard in every engine here, so this is no longer a differentiator.

Prefix caching and RadixAttention. If many requests share a prefix (a system prompt, a tool schema, earlier turns of a chat), you can reuse its KV blocks instead of recomputing them. SGLang's RadixAttention keeps cached prefixes in a radix tree with LRU eviction; the SGLang paper reported up to 6.4 times higher throughput on agent, few-shot, JSON, RAG and multi-turn workloads, from RadixAttention combined with its faster structured-output decoding. vLLM has automatic prefix caching too, and TensorRT-LLM has KV block reuse. The difference is now one of degree and of routing: in a multi-replica fleet, the gain only shows up if the router sends a request to the replica that already holds its prefix, which is what Dynamo's KV-aware router and llm-d's scheduler do.

This matters for agents in particular. An agent loop re-sends a long, mostly identical context on every step. A benchmark of unique random prompts measures none of that reuse, which is one reason published numbers rarely match production.

Does speculative decoding make LLM inference faster?

Sometimes, and the engines are now honest about when. Speculative decoding drafts several tokens cheaply, then has the big model verify them in one forward pass; when the drafts are accepted you get several tokens for roughly the price of one step. EAGLE-3 reports speedups of up to 6.5 times, but the same abstract reports a much smaller 1.38 times throughput gain in SGLang at batch size 64. That gap is the whole story: at low concurrency the GPU has spare compute, so verifying extra tokens is nearly free; at high concurrency the compute is already busy serving other requests.

The vLLM speculative decoding docs say the same thing: it is aimed at medium-to-low QPS, memory-bound workloads, model-based methods such as EAGLE and MTP give the largest latency reduction, and n-gram methods give modest gains without adding load at peak. TensorRT-LLM's 1.4 release candidates deprecate its n-gram and user-provided modes (v1.4.0rc2 notes), which tells you where NVIDIA thinks the value is. My rule: turn it on for interactive, latency-sensitive endpoints, measure inter-token latency and throughput at your real concurrency, and keep it off for batch jobs.

vLLM vs SGLang vs TensorRT-LLM: what do published benchmarks show?

The useful published numbers fall into three groups, and they answer different questions.

Third-party single-GPU comparisons. The cleanest recent head-to-head I found is Spheron's H100 test: Llama 3.3 70B in FP8 on one H100 SXM, 512 input and 256 output tokens on average, vLLM v0.18.0, SGLang v0.5.9 and TensorRT-LLM v1.2.0. At 100 concurrent requests it measured 2,400 output tokens per second for vLLM, 2,460 for SGLang and 2,780 for TensorRT-LLM, with p95 TTFT of 1,450, 1,380 and 1,280 ms. It also noted that the TensorRT-LLM engine build took about 28 minutes against roughly a minute of startup for the other two. Note what that test excluded: its prompts were unique, so SGLang's prefix caching had nothing to reuse. Spheron sells GPU hours, so read it as one data point, not a verdict.

Project-published scale results. These show what each engine can do with expert tuning on big MoE models. The SGLang team's 96-GPU DeepSeek deployment with prefill/decode disaggregation and wide expert parallelism reported 52.3k input and 22.3k output tokens per second per 8-GPU H100 node. vLLM's wide-EP write-up reported a sustained 2.2k tokens per second per H200 on DeepSeek-style models, up from about 1.5k. Dynamo's README cites 7 times higher throughput per GPU for DeepSeek R1 on GB200 NVL72 with Dynamo versus B200 without, measured by InferenceX. These are not comparable to each other (different hardware, models and dates), and each team chose its own best case.

Continuous public benchmarks. SemiAnalysis's InferenceX (formerly InferenceMAX) runs vLLM, SGLang and TensorRT-LLM continuously across NVIDIA Hopper and Blackwell, AMD MI300X to MI355X and TPUs, with results on a public dashboard. If you need a cross-engine number for a specific GPU and model, start there, because the configs are in the repo and you can rerun them. MLPerf Inference v6.0, published 1 April 2026 with 24 submitters, added GPT-OSS 120B and a DeepSeek-R1 interactive scenario that allows speculative decoding, but it is a hardware benchmark first and does not compare engines.

My reading of all of it: on NVIDIA hardware, a well-tuned TensorRT-LLM usually posts the highest per-GPU numbers, SGLang wins when prefixes repeat or when you need its large-scale MoE recipes, and vLLM is close behind on most single-node workloads while running on more hardware than either. The spread in the figure is 16 percent. Choosing the wrong tensor-parallel degree, KV cache dtype or max batch size can cost you more than that.

Where do Dynamo and llm-d fit?

They are not engines. Dynamo describes itself as an orchestration layer that turns vLLM, SGLang or TensorRT-LLM into a coordinated multi-node system: disaggregated prefill and decode, KV-aware routing, multi-tier KV caching and autoscaling. Version 1.6.0 moved its bundled engines to SGLang v0.5.21, TensorRT-LLM v1.3.0rc29 and vLLM v0.31.0, which is a useful reminder that Dynamo pins engine versions and you inherit its pinning. llm-d does a similar job natively on Kubernetes through the Gateway API inference extension, and it reached v1.0.0 on 20 October 2026 with a stable router API and a DisaggregatedSet for rolling prefill and decode replicas together.

You need one of these when you have more than a handful of replicas, long shared contexts worth routing on, or prompts long enough that prefill and decode want different GPU pools. Below that, a plain Deployment with a queue-depth autoscaler is less to operate. This is where platform discipline earns its keep: version pinning, staged rollouts and a benchmark gate in CI matter more to your p99 than which engine logo is on the pod.

Here is how I run a fair first comparison on one GPU node: same model, same port layout, both servers pinned, and the same client against both.

#!/usr/bin/env bash
# Side-by-side vLLM and SGLang on one node, same model, pinned images.
set -euo pipefail
MODEL=Qwen/Qwen3-8B
HF_CACHE=$HOME/.cache/huggingface

# vLLM v0.32.0 (OpenAI-compatible server on :8000), GPU 0
docker run -d --name vllm --gpus '"device=0"' --ipc=host \
  -v "$HF_CACHE":/root/.cache/huggingface -p 8000:8000 \
  vllm/vllm-openai:v0.32.0 \
  --model "$MODEL" --max-model-len 8192 --gpu-memory-utilization 0.90

# SGLang v0.5.22 (OpenAI-compatible server on :30000), GPU 1
docker run -d --name sglang --gpus '"device=1"' --ipc=host \
  -v "$HF_CACHE":/root/.cache/huggingface -p 30000:30000 \
  lmsysorg/sglang:v0.5.22 \
  python3 -m sglang.launch_server --model-path "$MODEL" \
    --context-length 8192 --host 0.0.0.0 --port 30000

# Then point the same client (vllm bench serve or GuideLLM) at both ports,
# with identical prompts, concurrency levels and output lengths.

Which LLM inference engine should you choose?

• Default for a team on GPUs: vLLM. It runs on NVIDIA, AMD, TPU and more, supports new models quickly, and has the largest operator community, so your failure modes are already in someone's GitHub issue. Deploying it is covered step by step in How to Deploy vLLM on Kubernetes with GPUs.
• Agents, RAG and multi-turn chat with heavy prefix reuse, or very large MoE models at scale: SGLang. Benchmark it against vLLM on your own traces, not random prompts, because its advantage is in reuse.
• NVIDIA-only fleet chasing cost per token, with engineers to tune it: TensorRT-LLM, ideally behind Dynamo. Budget for slower upgrades, RC-heavy releases and engine build time.
• Laptops, CPUs, edge boxes, Apple Silicon, single-user tools: llama.cpp (or Ollama on top of it). See Running LLMs Locally in 2026.
• Still on TGI: migrate to vLLM or SGLang.
• Low or spiky traffic, possibly none of them: an API can be cheaper than an idle GPU, which I worked through in Self-Hosting LLMs vs API Costs in 2026.

FAQ

Is SGLang faster than vLLM?

On workloads with lots of shared prefixes (agents, few-shot prompts, multi-turn chat) SGLang often is, thanks to RadixAttention. On unique-prompt workloads, independent tests such as Spheron's H100 run show them within a few percent of each other. Benchmark both on your own traffic before switching.

Is TensorRT-LLM worth the extra effort?

If you run only NVIDIA GPUs at a scale where a 10 to 20 percent cost-per-token gain pays for an engineer, often yes. If you run a few GPUs, change models frequently or need AMD support, vLLM or SGLang is less to operate. TensorRT-LLM's last stable release is 1.2.1 from April 2026, with 1.4 still in release candidates.

Is Hugging Face TGI still maintained?

Only minimally. Its README says it is in maintenance mode, accepting only minor fixes, and recommends vLLM, SGLang, llama.cpp or MLX instead. The repository is archived and the last release was v3.3.7 in December 2025.

Do I need NVIDIA Dynamo or llm-d?

Not for a single node or a few replicas. They pay off when you need KV-aware routing across many replicas, disaggregated prefill and decode, or coordinated multi-node rollouts. Both run vLLM or SGLang underneath, so you can start without them and add one later.

This quarter I would do three things: move anything still on TGI to vLLM, pin engine image tags the same way I pin base images, and put a short concurrency-sweep benchmark into the release pipeline so an engine upgrade cannot quietly add 200 ms to p99 TTFT. Then, and only then, I would try SGLang on the prefix-heavy agent traffic where it has a real chance to win.

Related: LLM Inference in 2026: Prefill Is a Different Machine Now, and the KV Cache Has a Disk Tier, GPU Clouds in 2026: H100s at $2.59 an Hour and SRE for LLM Apps in 2026: SLOs for TTFT, Provider Outages and Evals.