How to Calculate VRAM and KV Cache for LLM Inference (2026)

This guide is for engineers who need to answer "will this model fit, and how many users will it serve?" before renting a GPU, not after. By the end you will have a repeatable method: read a model's config.json, compute weight memory and KV cache per token by hand, scale it by context length, concurrency and tensor parallelism, run a small Python calculator that handles GQA, MLA and sliding-window models, and confirm the numbers against vLLM's startup log. Everything here was checked against vLLM v0.32.0 (the current stable release on 10 November 2026, per the releases page) and real Hugging Face configs for Qwen3, gpt-oss and DeepSeek-V3. You need curl, jq and Python 3.9 or later; a GPU is only needed for the last step. For the background on why the KV cache, not the weights, usually decides which GPU you need, read the companion explainer, LLM VRAM Requirements in 2026.

Step 1: Read the model's config.json

Every number you need for the KV cache lives in the model's config.json. Fetch it directly; for ungated models no token is needed.

curl -sL https://huggingface.co/Qwen/Qwen3-32B/resolve/main/config.json \
  | jq '{num_hidden_layers, num_attention_heads, num_key_value_heads,
         head_dim, hidden_size, max_position_embeddings, torch_dtype}'

Output, as I ran it:

{
  "num_hidden_layers": 64,
  "num_attention_heads": 64,
  "num_key_value_heads": 8,
  "head_dim": 128,
  "hidden_size": 5120,
  "max_position_embeddings": 40960,
  "torch_dtype": "bfloat16"
}

What each field means for memory:

• num_hidden_layers: every layer stores its own K and V.
• num_key_value_heads: the number of K/V heads. If it is smaller than num_attention_heads, the model uses grouped-query attention and the cache shrinks by that ratio (here 64 / 8 = 8x).
• head_dim: dimensions per head. If the field is missing, it is hidden_size / num_attention_heads. Do not assume that rule when the field exists: Qwen3-32B has hidden_size 5120 and 64 heads, which would give 80, but its real head_dim is 128.
• max_position_embeddings: the positional limit, not necessarily the recommended context. Qwen3's model card says 32,768 tokens natively and 131,072 with YaRN.

For multimodal models, look inside text_config. For DeepSeek-style MLA models, the fields that matter are different:

curl -sL https://huggingface.co/deepseek-ai/DeepSeek-V3/resolve/main/config.json \
  | jq '{num_hidden_layers, kv_lora_rank, qk_rope_head_dim, qk_nope_head_dim, v_head_dim}'

{
  "num_hidden_layers": 61,
  "kv_lora_rank": 512,
  "qk_rope_head_dim": 64,
  "qk_nope_head_dim": 128,
  "v_head_dim": 128
}

Step 2: Calculate weight memory

Weight memory is parameters times bytes per parameter. Get the exact parameter count from the Hub API rather than trusting the number in the model name:

curl -s https://huggingface.co/api/models/Qwen/Qwen3-32B | jq '.safetensors'

{
  "parameters": {
    "BF16": 32762123264
  },
  "total": 32762123264
}

So Qwen3-32B is 32,762,123,264 parameters. At BF16 (2 bytes) that is 65.5 GB, or 61.02 GiB. At FP8 it is 30.5 GiB. Two rules:

• Mixture-of-experts models count every expert. gpt-oss-120b activates 5.1B parameters per token but all 116.8B are resident.
• For pre-quantized checkpoints, use file size. Mixed formats such as MXFP4 or AWQ store scales and some layers at higher precision, so parameters times 0.5 undercounts. Sum the safetensors files instead:

curl -s https://huggingface.co/api/models/openai/gpt-oss-120b/tree/main \
  | jq '[.[] | select(.path | endswith(".safetensors")) | .size] | add / 1073741824'
# 60.76... (GiB)

Step 3: Calculate KV cache per token

For a standard GQA model:

kv_bytes_per_token = 2 x num_hidden_layers x num_key_value_heads x head_dim x kv_bytes
# 2 = one K and one V; kv_bytes = 2 for BF16/FP16, 1 for FP8

Qwen3-32B at BF16: 2 x 64 x 8 x 128 x 2 = 262,144 bytes (256 KiB) per token.

MLA models (DeepSeek-V2, V3 and derivatives) cache one compressed latent of size kv_lora_rank plus one rotary key of size qk_rope_head_dim per layer, with no per-head multiplication, as introduced in the DeepSeek-V2 paper:

kv_bytes_per_token = num_hidden_layers x (kv_lora_rank + qk_rope_head_dim) x kv_bytes
DeepSeek-V3: 61 x (512 + 64) x 2 = 70,272 bytes (68.6 KiB)

vLLM sizes the MLA cache with exactly that kv_lora_rank + qk_rope_head_dim width in its DeepSeek model code.

Sliding-window layers cap how many tokens a layer stores. gpt-oss-120b's config lists 36 layer_types, alternating sliding_attention and full_attention, with sliding_window set to 128. Only the 18 full layers grow with context: 2 x 18 x 8 x 64 x 2 = 36,864 bytes per token, plus a fixed 4.5 MiB per sequence for the sliding layers.

Step 4: Multiply by context length and concurrency

kv_total = kv_bytes_per_token x tokens_per_sequence x concurrent_sequences

Qwen3-32B, 8 users at 32,768 tokens each, BF16 cache: 262,144 x 32,768 x 8 = 64 GiB. That is more than the 61 GiB of weights. Switching the cache to FP8 halves it to 32 GiB.

Two practical notes. First, tokens_per_sequence is prompt plus generated output, so a 30K prompt with a 2K answer is a 32K sequence. Second, vLLM's PagedAttention allocates blocks as tokens arrive, so the real cache is shared: eight users averaging 8K tokens need a quarter of the worst case. Size for the worst case you will admit, then let the scheduler handle the average.

Step 5: Run a Python VRAM calculator

This script implements Steps 2 to 4 plus the tensor-parallel rules from Step 6. It uses only the standard library.

#!/usr/bin/env python3
"""Estimate per-GPU inference VRAM: weights + KV cache + overhead.

Reads a Hugging Face config.json. Python 3.9+, standard library only.
Example:
  python3 vram_calc.py Qwen3-32B.json --params 32762123264 --wbytes 2 \
      --ctx 32768 --seqs 8 --kvbytes 2 --tp 1
"""
import argparse
import json

GIB = 1024 ** 3


def kv_per_token(cfg, kvbytes, tp, ctx):
    """Return (KV bytes per token for the model, KV bytes per token per GPU)."""
    layers = cfg["num_hidden_layers"]

    # MLA (DeepSeek V2/V3): one compressed latent + one rope key per layer.
    # vLLM treats it as a single KV head, so TP does NOT split it.
    if cfg.get("kv_lora_rank"):
        b = layers * (cfg["kv_lora_rank"] + cfg["qk_rope_head_dim"]) * kvbytes
        return b, b

    heads = cfg["num_attention_heads"]
    kv_heads = cfg.get("num_key_value_heads") or heads  # GQA when < heads
    head_dim = cfg.get("head_dim") or cfg["hidden_size"] // heads
    per_layer = 2 * kv_heads * head_dim * kvbytes      # 2 = K and V

    # Sliding-window layers (e.g. gpt-oss) never hold more than `window` tokens.
    types = cfg.get("layer_types")
    window = cfg.get("sliding_window")
    if types and window:
        full = types.count("full_attention")
        sliding = len(types) - full
        avg_layers = full + sliding * min(window, ctx) / ctx
    else:
        avg_layers = layers

    b = per_layer * avg_layers
    # TP splits KV heads across GPUs; beyond kv_heads they are replicated.
    return b, b / min(tp, kv_heads)


def main():
    p = argparse.ArgumentParser()
    p.add_argument("config")
    p.add_argument("--params", type=float, help="total parameters (MoE: all experts)")
    p.add_argument("--wbytes", type=float, default=2.0, help="2 bf16, 1 fp8, ~0.55 4-bit")
    p.add_argument("--weights-gib", type=float, help="override: checkpoint size in GiB")
    p.add_argument("--ctx", type=int, default=8192, help="tokens per sequence")
    p.add_argument("--seqs", type=int, default=1, help="concurrent sequences")
    p.add_argument("--kvbytes", type=float, default=2.0, help="2 bf16, 1 fp8")
    p.add_argument("--tp", type=int, default=1, help="tensor-parallel GPUs")
    p.add_argument("--overhead", type=float, default=0.10,
                   help="activations, CUDA graphs, fragmentation")
    a = p.parse_args()

    cfg = json.load(open(a.config))
    cfg = cfg.get("text_config", cfg)  # multimodal configs nest the LM config

    weights = a.weights_gib * GIB if a.weights_gib else a.params * a.wbytes
    tok, tok_gpu = kv_per_token(cfg, a.kvbytes, a.tp, a.ctx)
    kv = tok * a.ctx * a.seqs
    gpu = (weights / a.tp + tok_gpu * a.ctx * a.seqs) * (1 + a.overhead)

    print(f"KV per token     : {tok:,.0f} bytes ({tok / 1024:.1f} KiB)")
    print(f"Weights          : {weights / GIB:8.2f} GiB")
    print(f"KV cache         : {kv / GIB:8.2f} GiB ({a.seqs} x {a.ctx:,} tokens)")
    print(f"Per GPU, TP={a.tp:<4} : {gpu / GIB:8.2f} GiB incl. {a.overhead:.0%} overhead")


if __name__ == "__main__":
    main()

Download configs and run it. These are the real outputs from my machine:

curl -sL https://huggingface.co/Qwen/Qwen3-32B/resolve/main/config.json -o Qwen3-32B.json
python3 vram_calc.py Qwen3-32B.json --params 32762123264 --ctx 32768 --seqs 8

KV per token     : 262,144 bytes (256.0 KiB)
Weights          :    61.02 GiB
KV cache         :    64.00 GiB (8 x 32,768 tokens)
Per GPU, TP=1    :   137.53 GiB incl. 10% overhead

The same workload with FP8 weights and FP8 cache:

python3 vram_calc.py Qwen3-32B.json --params 32762123264 --wbytes 1 --kvbytes 1 --ctx 32768 --seqs 8

KV per token     : 131,072 bytes (128.0 KiB)
Weights          :    30.51 GiB
KV cache         :    32.00 GiB (8 x 32,768 tokens)
Per GPU, TP=1    :    68.76 GiB incl. 10% overhead

And gpt-oss-120b with four 128K-token sequences, using the checkpoint size from Step 2:

curl -sL https://huggingface.co/openai/gpt-oss-120b/resolve/main/config.json -o gpt-oss-120b.json
python3 vram_calc.py gpt-oss-120b.json --weights-gib 60.77 --ctx 131072 --seqs 4

KV per token     : 36,900 bytes (36.0 KiB)
Weights          :    60.77 GiB
KV cache         :    18.02 GiB (4 x 131,072 tokens)
Per GPU, TP=1    :    86.67 GiB incl. 10% overhead

The model card says gpt-oss-120b fits on "a single 80GB GPU", and it does, but four full-length conversations at a BF16 cache do not: an 80 GB card is only about 74.5 GiB. The 10 percent overhead is a planning margin, not a measurement; Step 7 replaces it with real numbers.

Step 6: Account for tensor parallelism

With tensor parallelism (TP), weights divide evenly by the TP size. The KV cache divides by KV head, with two exceptions taken straight from vLLM's get_num_kv_heads:

• More GPUs than KV heads: heads are replicated so each GPU holds at least one. Qwen3-30B-A3B has 4 KV heads, so at TP=8 each GPU still holds one full head and the cache costs twice what naive division suggests.
• MLA: returns one KV head, so every GPU stores the whole latent cache.

python3 vram_calc.py DeepSeek-V3.json --params 684489845504 --wbytes 1 --ctx 131072 --tp 8

KV per token     : 70,272 bytes (68.6 KiB)
Weights          :   637.48 GiB
KV cache         :     8.58 GiB (1 x 131,072 tokens)
Per GPU, TP=8    :    97.09 GiB incl. 10% overhead

97 GiB per GPU rules out 8 x H100 (about 74.5 GiB each) and fits 8 x H200, which NVIDIA lists at 141 GB (about 131 GiB) each. The parameter count includes DeepSeek-V3's multi-token prediction module, so this slightly overestimates; that is the safe direction to be wrong in.

Step 7: Check your numbers against vLLM's logs

Now run it on real hardware and compare. Pin the version, because memory defaults move between releases: in v0.32.0, gpu_memory_utilization defaults to 0.92, not the 0.9 older tutorials quote.

# vLLM v0.32.0, one 80 GB GPU, FP8 weights and FP8 KV cache
docker run --gpus all --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:v0.32.0 \
  --model Qwen/Qwen3-32B-FP8 \
  --max-model-len 32768 \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.92 \
  --max-num-seqs 32 2>&1 | grep -E "Model loading took|Available KV cache memory|KV cache size"

The three lines to read are these message formats from the v0.32.0 source (the values are filled in at runtime):

Model loading took <X> GiB memory and <S> seconds
Available KV cache memory: <Y> GiB
GPU KV cache size: <N> tokens, Maximum concurrency for 32,768 tokens per request: <C>x

Check them in this order:

1. Model loading took should match Step 2 within a GiB or so. Much higher usually means a different precision than you assumed.
2. Available KV cache memory is total memory times gpu_memory_utilization, minus weights, activations and CUDA graph memory. Divide it by your per-token number from Step 3. The result should be close to <N> tokens.
3. Maximum concurrency is <N> divided by --max-model-len. If it is below 1, not even one full-length request fits.

The ratio between your calculated overhead and the real one is the number to keep. Feed it back into --overhead for the next model on the same GPU and the calculator becomes accurate for your fleet. If you deploy this on a cluster, the same flags go into the container args in How to Deploy vLLM on Kubernetes with GPUs.

Troubleshooting

Symptom	Likely cause	Fix
ValueError: No available memory for the cache blocks	Weights plus profiled activations exceed the gpu_memory_utilization budget	Use FP8 or 4-bit weights, raise --gpu-memory-utilization slightly, or add GPUs with --tensor-parallel-size
To serve at least one request with the model's max seq len (...) ... GiB KV cache is needed, which is larger than the available KV cache memory	Default max_model_len from config is longer than the cache can hold	Set --max-model-len to the estimate the error prints, or use --kv-cache-dtype fp8
torch.OutOfMemoryError: CUDA out of memory during startup after weights load	CUDA graph capture or another process on the GPU	Check nvidia-smi for other processes, try --enforce-eager to confirm, then lower --max-num-seqs
Concurrency in the log is far below your estimate	You computed per-token KV with hidden_size / heads instead of the real head_dim, or forgot replication when TP exceeds KV heads	Recompute from head_dim and num_key_value_heads; use TP no larger than the KV head count
Model "fits" by your maths but not in practice	GB versus GiB: 80 GB is about 74.5 GiB	Do every calculation in GiB
Your numbers are far too high for a DeepSeek model	Calculated with 128 heads instead of the MLA latent	Use num_hidden_layers x (kv_lora_rank + qk_rope_head_dim) x bytes
KeyError: 'num_hidden_layers' in the script	Multimodal config nests the language model	The script reads text_config; for other layouts, point it at the nested dict
Two vLLM instances on one GPU fail to start	Each instance's gpu_memory_utilization is a per-instance fraction	Set both so the sum stays under 1.0, for example 0.45 each
Quality drops after enabling FP8 KV cache	Uncalibrated FP8 scales default to 1.0	Use calibrated scales via llm-compressor as the quantized KV cache docs recommend, and rerun your evals

The first two error messages are quoted from kv_cache_utils.py in v0.32.0. The vLLM conserving memory page covers the remaining knobs, including smaller CUDA graph capture sizes.

FAQ

What is the formula for KV cache size?

For GQA and standard attention: 2 x layers x KV heads x head dimension x bytes per element, per token. Multiply by tokens per sequence and by concurrent sequences. For MLA models, use layers x (kv_lora_rank + qk_rope_head_dim) x bytes instead.

Does tensor parallelism reduce KV cache per GPU?

Yes for GQA models, by the TP size, until TP exceeds the number of KV heads; past that point heads are replicated. For MLA models such as DeepSeek-V3, no: vLLM stores the full latent cache on every GPU.

How do I set vLLM to use a fixed amount of KV cache?

vLLM v0.32.0 has a kv_cache_memory_bytes option (--kv-cache-memory-bytes on the CLI) that sets the KV cache per GPU in bytes and, when set, ignores gpu_memory_utilization. It is useful when sharing a GPU or when you want reproducible capacity across hosts.

Why does vLLM use almost all GPU memory even for a small model?

By design. It reserves gpu_memory_utilization of the card, 92 percent by default in v0.32.0, and gives everything after weights and activations to the KV cache. Lower the fraction if something else must share the GPU.

When I size a deployment, the first thing I check is num_key_value_heads and head_dim in the config, because those two fields explain most of the surprises, followed by the "Maximum concurrency" line in vLLM's log, because it is the only number that matters to users. I keep the measured overhead per GPU type in the same repository as the deployment manifests, so the next sizing exercise starts from data instead of a rule of thumb. The official references are the vLLM engine arguments, the quantized KV cache docs and the Hugging Face model configs.

Related: How to Deploy vLLM on Kubernetes with GPUs, How to Benchmark LLM Inference: TTFT, Throughput and Goodput and How to Install NVIDIA GPU Operator and Share GPUs.