LLM VRAM Requirements in 2026: Weights, KV Cache and Which GPU Fits

The question I get most from people about to rent or buy their first inference GPU is "how much VRAM do I need for this model?", and the usual answer, "two gigabytes per billion parameters", is right about the part that matters least once you have real users. That rule covers the weights. It says nothing about the KV cache, which grows with every token of context and every concurrent request, and on a busy server it is routinely bigger than the model itself. This post explains LLM VRAM requirements from first principles: what weights cost at each precision, the KV cache formula worked from real config.json files, how context length and batch size multiply it, what the 2026 GPUs actually hold, and when you need tensor parallelism. If you want the formulas as a script you can run, the companion guide, How to Calculate VRAM and KV Cache for LLM Inference, walks through it step by step.

How much VRAM does an LLM need? The three buckets

Inference memory on a GPU splits into three parts, and it helps to keep them separate in your head:

1. Weights. Parameters times bytes per parameter. Fixed for a given checkpoint and precision.
2. KV cache. The keys and values every attention layer stores for every token already processed, so the model does not recompute them on each new token. Grows linearly with tokens in flight.
3. Overhead. Activations for the current batch, CUDA graphs, the CUDA context itself, the framework's allocator slack. Usually a few GiB, and the part people forget until the first CUDA out of memory.

Serving engines make this split explicit. vLLM loads the weights, runs a profiling pass to measure peak activation memory, and then hands everything left within its budget to the KV cache. That budget is gpu_memory_utilization, which defaults to 0.92 in vLLM v0.32.0, the current stable release on the releases page. So on a GPU, the honest sizing question is not "do the weights fit?" but "after the weights fit, how many tokens of KV cache are left?".

One unit trap before the numbers: GPU vendors quote decimal gigabytes, frameworks report binary GiB. An "80 GB" H100 is about 74.5 GiB. That 7 percent decides more fits than you would expect.

Weight memory by precision: BF16, FP8 and 4-bit

Weights are the easy bucket. Bytes per parameter is 2 for BF16 or FP16, 1 for FP8 or INT8, and roughly 0.5 for 4-bit formats (a little more in practice, because quantized formats store scales alongside the values). The parameter counts below come from the Hugging Face model API's safetensors metadata, and the checkpoint sizes from the repository file listing, both of which I queried directly.

Model	Parameters	Shipped precision	Checkpoint on disk	Same model at FP8	Source
Qwen3-8B	8.19B	BF16	15.3 GiB	~7.6 GiB	Hugging Face
Qwen3-32B	32.76B	BF16	61.0 GiB	~30.5 GiB	Hugging Face
gpt-oss-120b	116.8B (5.1B active)	MXFP4 MoE weights	60.8 GiB	n/a	Hugging Face
DeepSeek-V3	684.5B (incl. MTP module)	FP8	641.3 GiB	already FP8	Hugging Face

Two things fall out of this table. First, mixture-of-experts models do not get a discount on memory: gpt-oss-120b only activates 5.1B parameters per token, but all 117B must be resident. Its model card says the MXFP4 quantization of the MoE weights is what lets it run "on a single 80GB GPU (like NVIDIA H100 or AMD MI300X)". Second, precision is the biggest lever you have on the weights bucket. Halving bytes per parameter frees exactly as much memory as the weights occupied at the lower precision, and that memory goes straight to the KV cache. The trade-offs between FP8, AWQ, GPTQ and GGUF are a topic of their own, covered in LLM quantization explained.

The KV cache formula, and why context length is the real cost

For a standard transformer with grouped-query attention, the KV cache per token is:

kv_bytes_per_token = 2 x num_hidden_layers x num_key_value_heads x head_dim x bytes_per_element
                     ^ one for K, one for V
total_kv_bytes     = kv_bytes_per_token x tokens_per_sequence x concurrent_sequences

Every term on the right of the first line is in the model's config.json. Qwen3-32B's config has 64 layers, 64 attention heads, 8 KV heads and a head dimension of 128. At BF16 that is 2 x 64 x 8 x 128 x 2 = 262,144 bytes, exactly 256 KiB per token. Qwen3's model card gives a native context of 32,768 tokens, so one full-length sequence needs 8 GiB of KV cache. Eight of them need 64 GiB, which is more than the 61 GiB of weights.

The num_key_value_heads term is where architecture matters. With classic multi-head attention, every query head has its own K and V; with grouped-query attention (GQA), groups of query heads share one K/V pair. Qwen3-32B shares each KV head among 8 query heads, so its cache is one eighth of what a full multi-head version would need: 256 KiB per token instead of 2 MiB.

DeepSeek went further with multi-head latent attention (MLA), introduced in the DeepSeek-V2 paper, which reports that it "reduces the KV cache by 93.3%" against their previous model. Instead of per-head keys and values, MLA caches one compressed latent vector per layer plus a small rotary key. In DeepSeek-V3's config, kv_lora_rank is 512 and qk_rope_head_dim is 64, across 61 layers: 61 x (512 + 64) x 2 bytes = 70,272 bytes, about 68.6 KiB per token, the same latent-plus-rope sizing vLLM uses in its DeepSeek implementation. If the same 128 heads cached full keys (192 dimensions) and values (128) per head, it would be about 4.77 MiB per token. At 128K tokens that is the difference between 8.6 GiB and 610 GiB.

Model	Attention	Layers x KV heads x head dim	KV per token (BF16)	One 32K sequence	Source
Qwen3-8B	GQA	36 x 8 x 128	144 KiB	4.5 GiB	config.json
Qwen3-30B-A3B	GQA (MoE)	48 x 4 x 128	96 KiB	3.0 GiB	config.json
Qwen3-32B	GQA	64 x 8 x 128	256 KiB	8.0 GiB	config.json
gpt-oss-120b	GQA, half the layers sliding (128 tokens)	18 full layers x 8 x 64	36 KiB plus a fixed 4.5 MiB	1.13 GiB	config.json
DeepSeek-V3	MLA	61 x (512 + 64) latent	68.6 KiB	2.1 GiB	config.json

The gpt-oss row shows a third trick. Its layer_types list alternates sliding_attention and full_attention across 36 layers, with sliding_window set to 128, so half the layers never hold more than 128 tokens of keys and values regardless of context length. Only the 18 full-attention layers grow with the sequence.

Qwen3-30B-A3B, nearly the same size on disk as Qwen3-32B, needs well under half the KV cache per token: two models of equal size can differ by 2x in how many users one card serves.

Batch size and concurrency: where serving budgets actually go

A single user chatting with a local model rarely runs out of KV cache. A server does, because concurrency multiplies everything. Here is the budget for Qwen3-32B on one H100 SXM, which NVIDIA lists at 80 GB of HBM3:

• 80 GB is about 74.5 GiB; at vLLM's default 0.92 utilization, the budget is roughly 68.6 GiB.
• BF16 weights take 61 GiB, leaving about 7.5 GiB.
• Subtract a few GiB of activations and CUDA graphs and perhaps 4 to 5 GiB remain for the cache, which at 256 KiB per token is something like 16,000 to 20,000 tokens in total.

That is not even one full 32K-token sequence. The model "fits", and it is nearly useless as a server. Switch to FP8 weights and FP8 KV cache and the picture changes completely: weights drop to about 30.5 GiB, each token costs 128 KiB, and the same card holds on the order of 250,000 tokens of cache, enough for several long conversations at once. These are my estimates from the formula, not measurements; the exact figure depends on what the profiler finds, and the guide shows how to read it from vLLM's startup log.

vLLM's PagedAttention allocates the cache in fixed-size blocks as tokens arrive, so a short request only pays for what it uses and the "tokens in total" figure is shared across all requests in flight. The scheduler caps concurrency with max_num_seqs, which the v0.32.0 engine arguments default to 1,024 for the API server on H100 and H200 class GPUs and 256 on smaller cards. When the cache fills, requests queue or get preempted. That is why a serving cluster that looked fine in a single-user test falls over under load: the weights were the same, the KV cache was not. If you are measuring this, benchmark goodput, not just throughput.

GPU memory specs in 2026: RTX 4090 to MI355X

Capacity decides what fits. Bandwidth decides how fast it runs, because generating each token requires reading every active weight from memory at least once. A rough ceiling for single-sequence decode speed is bandwidth divided by the bytes of active weights: 3.35 TB/s over 65.5 GB of Qwen3-32B BF16 weights is about 51 tokens per second on an H100, before any real-world losses. Batching amortizes that read across many sequences, which is why servers batch.

GPU	Memory	Type	Bandwidth	Source
GeForce RTX 4090	24 GB	GDDR6X	1,008 GB/s	NVIDIA, Tom's Hardware
GeForce RTX 5090	32 GB	GDDR7, 512-bit	1,792 GB/s	NVIDIA, Tom's Hardware
L40S	48 GB	GDDR6 ECC	864 GB/s	NVIDIA
H100 SXM	80 GB	HBM3	3.35 TB/s	NVIDIA
H100 NVL	94 GB	HBM3	3.9 TB/s	NVIDIA
H200	141 GB	HBM3e	4.8 TB/s	NVIDIA
B200 (in DGX B200)	180 GB	HBM3e	8 TB/s	NVIDIA (1,440 GB and 64 TB/s across 8 GPUs)
Instinct MI300X	192 GB	HBM3	5.3 TB/s	AMD data sheet
Instinct MI355X	288 GB	HBM3E	8 TB/s	AMD

The consumer cards are the interesting row for most readers. A 32 GB RTX 5090 holds Qwen3-8B at BF16 with plenty of room for context, or a 32B model only at 4-bit, and its 1.8 TB/s is more than half an H100's bandwidth for a fraction of the price. The L40S has more capacity than either but less bandwidth than a 4090, which is why it suits large-batch or multi-tenant work better than low-latency single streams. If you are choosing between buying cards and renting, the self-hosting versus API break-even is driven by utilisation more than by any of these specs.

When one GPU is not enough: tensor parallelism and VRAM

Tensor parallelism (TP) splits each layer's matrices across GPUs, so the weights divide by the TP size. The KV cache divides too, by KV head: each GPU stores the heads it computes. There are two catches, both visible in vLLM's get_num_kv_heads:

• If TP is larger than the model's KV head count, heads are replicated so each GPU has at least one. Qwen3-30B-A3B has 4 KV heads, so at TP=8 every KV head lives on two GPUs and you pay for it twice.
• MLA models return one KV "head" regardless of TP, because the latent is shared. DeepSeek-V3's 68.6 KiB per token is stored in full on every GPU in the group.

The DeepSeek-V3 arithmetic shows why 8 x H100 is not enough for it: 641 GiB of FP8 weights against 8 x 74.5 GiB, about 596 GiB, before a single token of cache. Eight H200s give about 1,050 GiB, so each GPU carries roughly 80 GiB of weights and has room for a large cache. A single node of eight MI355Xs has more than twice that.

TP across PCIe works but every layer does an all-reduce, so it is much slower than over NVLink or Infinity Fabric. For models that fit on one GPU, running separate replicas usually beats TP on throughput; TP is for models that do not fit, or for latency when one sequence must be fast. Scheduling those GPUs on a cluster is its own problem, covered in GPUs on Kubernetes explained.

Here is how I would launch the FP8 Qwen3-32B case on one 80 GB GPU, with every memory knob explicit so nobody has to guess later:

# vLLM v0.32.0 (pin the image tag; defaults change between releases)
docker run --gpus all --ipc=host -p 8000:8000 \
  vllm/vllm-openai:v0.32.0 \
  --model Qwen/Qwen3-32B-FP8 \
  --max-model-len 32768 \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.92 \
  --max-num-seqs 32
# Read two startup lines: "Available KV cache memory: X GiB" and
# "GPU KV cache size: N tokens, Maximum concurrency for 32,768 tokens per request: Yx"

The quantized KV cache docs note that without calibration, FP8 scales default to 1.0; for quality-sensitive work, calibrated scales via llm-compressor are the recommended path.

FAQ

How much VRAM do I need for a 70B model?

At BF16, about 140 GB for the weights alone, so two 80 GB GPUs at minimum and realistically an H200 pair or more for useful context. At FP8 it is about 70 GB, which technically fits on one 80 GB card with very little KV cache left. At 4-bit, about 35 to 40 GB, which fits a 48 GB L40S or two consumer cards. Then add the KV cache from the model's own config.

How do I calculate KV cache size?

Multiply 2 (keys and values) by the number of layers, the number of KV heads, the head dimension and the bytes per element, all read from config.json. That gives bytes per token; multiply by context length and the number of concurrent sequences. MLA models such as DeepSeek-V3 use kv_lora_rank plus qk_rope_head_dim per layer instead.

Does FP8 KV cache hurt quality?

It halves cache memory compared with BF16, and vLLM supports it on CUDA and ROCm. The effect on output quality depends on the model and on whether scales are calibrated, so measure it with your own evals rather than assuming it is free.

Is memory bandwidth or capacity more important for LLM inference?

Capacity decides whether a model and its cache fit at all; bandwidth decides decode speed once it does. For a single user, bandwidth matters most. For a server with many users, capacity matters most, because it caps how many sequences you can batch.

This quarter, before anyone signs a GPU contract, I would write down the target model, the context length real users need and the concurrency at peak, run those three numbers through the KV cache formula, and size against the cache rather than the weights. Then I would confirm it with vLLM's startup log on one rented card, which costs less than a meeting about it. Capacity planning was a platform discipline long before GPUs were involved, and it still is.

Related: How to Deploy vLLM on Kubernetes with GPUs, LLM Inference Engines 2026: vLLM vs SGLang vs TensorRT-LLM vs Dynamo and GPU Clouds in 2026.