LLM Inference in 2026: Prefill Is a Different Machine Now, and the KV Cache Has a Disk Tier

I build infrastructure that runs AI agents, not infrastructure that runs the models the agents call. But every agent on my platform spends most of its wall-clock time waiting on an inference endpoint, and the bill for that endpoint is the largest line item in my customers' costs. So once a year I read the serving-systems literature properly, to know what I am waiting on and why it costs what it does. This is the 2026 read, from the engine release notes, the papers and the benchmark repos, as of 7 September.

The one-paragraph version: in 2024 the state of the art was one engine on one node doing continuous batching with paged attention. In 2026 it is a distributed system in which the two phases of a request run on different machines, the attention cache lives in a four-tier memory hierarchy that spans hosts, a router picks the replica that already has your prompt's prefix cached, and a rack of 72 GPUs acts as one machine for a model with 1.6 trillion parameters of which 49 billion are active per token. Every open-source engine now ships all of that. The rest of this post is the detail, and at the end the prices, which behaved in a way the "cost of intelligence is collapsing" story does not predict.

The two phases were always different workloads

An LLM request has a prefill phase, where the whole prompt is processed in one parallel pass to build the key-value cache, and a decode phase, where tokens are generated one at a time, each step reading the entire cache. Prefill is compute-bound: matrix multiplications over thousands of tokens at once. Decode is memory-bandwidth-bound: a few tokens of compute per step but the full cache streamed from HBM every step. Putting both on the same GPU means the decode steps of one request stall behind the prefill of another, which shows up to the user as a jitter in time-per-output-token, and it means the hardware is sized wrong for at least one of the phases at any moment.

DistServe (OSDI 2024) and Splitwise (ISCA 2024) made the academic case for running the phases on separate pools. Mooncake, Moonshot's serving platform for Kimi, made the production case and won best paper at FAST '25 for it: a KV-cache-centric design where the cache is a first-class distributed object moved between prefill and decode nodes over RDMA, with a transfer engine that their repo reports at 87 GB/s over four 200-gigabit links and 190 GB/s over eight 400-gigabit ones. Mooncake joined the PyTorch ecosystem in February.

In 2026 the idea is table stakes. Here is the disaggregated launch in vLLM, from the current docs, where a worker's role is set by a connector config and the KV blocks move over NVIDIA's NIXL transfer library:

# one of these per worker; kv_role is kv_producer (prefill),
# kv_consumer (decode) or kv_both
vllm serve $MODEL --kv-transfer-config \
  '{"kv_connector":"NixlConnector","kv_role":"kv_both",
    "kv_buffer_device":"cuda",
    "kv_connector_extra_config":{"backends":["UCX","GDS"]}}'

The same flag family takes LMCacheConnectorV1, MooncakeConnector, a ROCm connector for AMD, and a MultiConnector that chains them. vLLM 0.28.0, released 26 August, goes further and splits a third phase out for multimodal models: encoder, prefill and decode each on their own workers. SGLang, TensorRT-LLM via Dynamo, and llm-d all have equivalents. If you are running a single engine on a single node in 2026, you are running the 2024 architecture, and it is fine for a lab and wrong for a product.

The cache became a storage system

Once the KV cache is an object that moves between machines, it can also be kept. A multi-turn agent conversation, a long system prompt shared by thousands of users, a document every request re-reads: all of these have a prefix whose cache was computed once already. Recomputing it is the single largest avoidable cost in production serving, which is why prefix caching is now on by default in every engine and, more tellingly, is now priced. DeepSeek charges $0.014 per million cache-hit input tokens on V4-Flash against $0.44 for a miss, a 30× difference. OpenAI's cached input is a tenth of the uncached price. When a provider passes a discount that steep through to the customer, you can infer the cost structure.

The engineering is a four-tier hierarchy. SGLang's RadixAttention keeps the prefix tree in GPU memory. LMCache extends it to CPU RAM, local disk, Redis-compatible stores and S3, and its underlying papers (CacheGen at SIGCOMM 2024, CacheBlend at EuroSys 2025) are about compressing the cache for transport and re-using it even when the prefix is not an exact match. NVIDIA's Dynamo has a KV block manager with GPU, CPU, SSD and remote tiers. vLLM 0.28 added its own disk tier and a CPU layout that is independent of the tensor-parallel configuration, so an offloaded block can be read back by a replica with a different parallelism. The offloading tier is one flag:

vllm serve $MODEL --kv-transfer-config \
  '{"kv_connector":"OffloadingConnector","kv_role":"kv_both",
    "kv_connector_extra_config":{"block_size":64,"cpu_bytes_to_use":1000000000}}'

If this sounds familiar to readers of this blog, it should. Snapshot-restore of a microVM, which is how I boot in 179 milliseconds, is the same idea one layer down: never recompute state you already have on disk, page it in on demand, and make the cache key content-addressed so identical prefixes share. The KV cache tier is userfaultfd for transformers. The routing consequence is also the same: the Gateway API Inference Extension, which reached v1 in September 2025 and is at 1.6 this month, routes on which replica already has the prefix, the way my scheduler routes to the host that already has the snapshot.

Smaller numbers, sparser attention, wider experts

Three other techniques matured this year and all three change what the hardware has to do.

Four-bit is the production format on Blackwell. NVIDIA's NVFP4 stores values in 4 bits with an 8-bit scale per 16-value block and a 32-bit scale per tensor, about 4.5 bits per weight, and their own measurement puts accuracy loss on DeepSeek-R1 at 1% or less against FP8. OpenAI's gpt-oss models ship with MXFP4 expert weights; DeepSeek-V4-Pro ships its expert parameters in FP4 and most of the rest in FP8. The point is not the memory saving, though halving 1.6 trillion parameters matters; it is that the decode phase is bandwidth-bound, so halving the bytes per parameter is close to doubling tokens per second.

Attention got sparse, then compressed. DeepSeek's Sparse Attention in V3.2 (December 2025) uses a lightweight indexer to pick the top-k cache entries each query actually needs, so the cost of long context stops scaling with the whole context. GLM-5 adopted it. V4 in April added token-wise compression on top, and Qwen3.5 went a different route with a Gated DeltaNet hybrid. For serving, this is why vLLM 0.28 has a "sparse MLA" path and why FlashAttention-4 (arXiv:2603.05451, March 2026) matters: Tri Dao's group reports 1,613 teraflops on B200, 71% of peak, with the softmax exponentials computed by polynomial on the FMA units because the special-function units were the bottleneck. When the attention pattern is no longer dense, the kernel has to be rewritten, and it was.

Mixture-of-experts runs across the rack. Every large open model in the table below is an MoE with a few percent of parameters active per token. Serving that efficiently means spreading experts across many GPUs so each one holds few and stays busy: "wide expert parallelism", built on DeepEP, whose second version reports 90 GB/s dispatch over RDMA. On a GB200 or GB300 NVL72, 72 GPUs share one NVLink domain, so the rack, not the node, is the unit of deployment. This is the single largest reason Blackwell's per-token economics are not comparable to Hopper's: an H100 node cannot do this at all.

Speculation is built into the models. Draft-based speculative decoding (Medusa, EAGLE-3) used to require training a separate head. DeepSeek V3 shipped a multi-token-prediction head in the base model; now GLM-5 and others do too, and turning it on is a serve-time flag from the model card:

vllm serve zai-org/GLM-5 \
  --speculative-config.method mtp \
  --speculative-config.num_speculative_tokens 3

The engines, and the hardware they run on

Engine	Latest	Stars	What distinguishes it in 2026
vLLM	0.28.0, 26 Aug 2026	91k	Default choice; connector ecosystem for KV transfer; E/P/D split
SGLang	0.5.19, 4 Sep 2026	36k	RadixAttention; README claims over 400,000 GPUs; xAI, Cursor, Baseten listed as users
TensorRT-LLM	1.3 rc	15k	Now PyTorch-native; the fastest path on NVIDIA when paired with Dynamo
NVIDIA Dynamo	1.4.2	8k	Router, SLA planner, KV block manager, NIXL; 1.0 in March
llm-d	0.9.0	4k	CNCF sandbox since March; vLLM on Kubernetes with "well-lit paths"
Ollama	0.34.0	180k	Local; now has an MLX engine beside llama.cpp
Text Generation Inference	3.3.5	11k	Maintenance mode; README points you at vLLM and SGLang

Hugging Face putting its own server into maintenance mode and recommending the competition is the cleanest signal in that table that the engine layer has consolidated. The layer above it, orchestration on Kubernetes, is where the current competition is: Dynamo, llm-d, ByteDance's AIBrix and the Gateway API extension all implement the router-plus-planner half of the diagram, and the difference between them is mostly which vendor's support contract you want.

On hardware, MLPerf Inference v6.0 in April added DeepSeek-R1 and gpt-oss-120b as benchmarks, which is the first time the standard suite reflects what people actually serve. The headline: 2.49 million tokens per second on DeepSeek-R1 offline across 288 Blackwell Ultra GPUs, and a per-GPU server-mode number on GB300 that went from 2,907 to 8,064 tokens per second between rounds through software alone. SemiAnalysis's continuously-run InferenceX benchmark reports GB200 with Dynamo, FP4 and MTP at $0.057 per million tokens on DeepSeek-R1 against $0.251 for the baseline configuration, and a 10× to 65× improvement in tokens per dollar from H100 to GB200 NVL72 depending on the interactivity target; note that it runs with prefix caching disabled, so it understates real workloads and overstates nothing. AMD published a rebuttal on like-for-like modes for MI355X that I could not load. Their Helios rack with MI455X, 432 GB of HBM4 per GPU, was announced in July for the second half. Google's TPU7x went GA in March. NVIDIA's Vera Rubin NVL72, with 288 GB HBM4 and 50 petaflops FP4 per GPU, is "in full production" and shipping this half per their August results, in which data centre revenue was $89 billion for the quarter. NVIDIA does not break inference out of that, and the frequently repeated claim that inference is now the majority of AI compute has, as far as I can find, no primary quantified source from any of the big three. Huang's actual words in May were that they are "growing share in inference very, very quickly", which is a different statement.

What you can serve, and what it costs

The open-weight models are the serving targets that make all of the above matter, and 2026 is the year they became genuinely large:

Model	Released	Total / active params	Context	Licence
DeepSeek-V4-Pro	Apr 2026	1.6T / 49B	1M	MIT
DeepSeek-V4-Flash	Apr 2026	284B / 13B	1M	MIT
Kimi K2.5	Jan 2026	1T / 32B	256K	modified MIT
GLM-5	Feb 2026	744B / 40B		MIT
Qwen3.5-397B-A17B	Feb 2026	397B / 17B	262K native	Apache 2.0
gpt-oss-120b	Aug 2025	117B / 5.1B	128K	Apache 2.0
Gemma 4	Apr 2026	2B to 31B; 26B-A4B MoE	128K to 256K	Apache 2.0
Llama 4 Maverick	Apr 2025	400B / 17B		Llama licence

Five of the eight most capable open models are from Chinese labs under MIT or Apache licences; Meta's largest Llama 4 variant is still, per Meta's own page, "training", sixteen months later. That is the serving landscape, and it is why the engines above have DeepSeek-specific code paths as their headline features.

And the prices, which is where I want to end, because the data disagrees with the story. The Epoch AI and a16z analyses that the cost of a fixed capability falls 10× or more per year remain true: what GPT-4 could do in 2023 costs a few cents per million tokens today from DeepSeek. But the list price of the frontier is not falling. From the providers' pages this week:

Provider, top tier	Input / output per million tokens
Anthropic, Fable 5.1	$10 / $50
Anthropic, Opus 5	$5 / $25
OpenAI, gpt-5.6 sol	$4 / $20
Google, Gemini 3.1 Pro	$2 / $12
DeepSeek, V4-Pro (peak)	$1.32 / $3.96
Google, Gemini 3.8 Flash	$0.75 / $3.75, rising to $1.50 / $7.50 on 1 Jan 2027

Google announcing a doubling of its mid-tier price for next year is the first frontier list-price increase I have seen a major provider pre-announce. The way to reconcile the two facts is that inference has become a commodity at fixed capability and a scarce good at the frontier, and the serving systems in this post are what set the floor on the commodity side. Google's own energy accounting puts a median Gemini text prompt at 0.24 watt-hours, down 33× in a year, with only 58% of that in the accelerator and the rest in host, idle and overhead; the systems work above is mostly about the 42%.

For my own platform the practical consequences are three. Agents should be built to hit the cache: stable system prompts, tools whose descriptions do not change per call, and conversation history that is appended rather than rewritten, because the price difference between a hit and a miss is now 10× to 30×. Latency budgets should separate time-to-first-token from time-per-token, because the providers now scale them on different machines and will price them differently soon. And the open-weight column is now good enough that "run it yourself on a rack" is a real option for a company with a rack, which for the first time in a while is a sentence an infrastructure engineer gets to say about the frontier of the field.

---

Related: Sandbox Creation Time Benchmarks Measure Different Things, I Measured Every Stage of My 179ms Firecracker Boot Path and Neuromorphic Computing in 2026.