LLM Quantization Explained (2026): FP8, NVFP4, GGUF, AWQ and GPTQ

Every few weeks someone asks me which file to download from a Hugging Face repo that lists twenty versions of the same model: Q4_K_M, Q5_K_M, Q8_0, AWQ, GPTQ, FP8, NVFP4. The honest answer is that LLM quantization is three separate decisions wearing one name: how many bits each number gets, which numbers you shrink (weights, activations, the KV cache), and which runtime has fast kernels for the result. This post explains each format in plain terms, shows the published evidence on how much quality you lose, and ends with a table of which format I would pick for llama.cpp, vLLM, TensorRT-LLM and MLX in late 2026.

What is LLM quantization, and what does it actually shrink?

A model checkpoint is mostly matrices of numbers. Training usually happens in BF16, 16 bits per number, so a 70-billion-parameter model is about 140 GB of weights before you allocate a single token of context. Quantization stores those numbers in fewer bits, plus a small number of scale factors that let you map the short codes back to roughly the right values.

Three things can be quantized, and the notation tells you which:

• Weights (the W in W4A16). Shrinking them cuts memory and, during decode, the bytes you have to stream from memory for every generated token. Decode on a single request is memory-bandwidth bound, so fewer bytes per weight is close to a direct speed-up there.
• Activations (the A). Quantizing them too, as in W8A8, lets the matrix multiply itself run on low-precision tensor cores. That matters most under heavy batching, where you are compute bound rather than bandwidth bound.
• The KV cache. This grows with context length and concurrency, not with parameter count, and on long-context serving it can be bigger than the weights. FP8 KV cache is now a standard switch in TensorRT-LLM and vLLM.

The arithmetic for weights is simple: parameters times bits divided by eight. A 70B model is 140 GB at 16 bits, 70 GB at 8 bits and roughly 39 GB at an effective 4.5 bits. If you want the full sizing method including the KV cache, LLM VRAM sizing explained works through it.

The number formats: BF16, FP8, INT8, INT4, MXFP4 and NVFP4

The formats differ in two ways: the shape of each number (integer or tiny float) and how scales are shared across a block of numbers. The smaller the block that shares a scale, the better it tracks local outliers, and the more bits you spend on scales.

Format	Bits per value	Scale shared by	Native tensor-core support	Source
BF16 / FP16	16	none	the baseline every runtime supports	llama.cpp README
FP8 E4M3 / E5M2	8	tensor, channel or block	compute capability 8.9+ (Ada, Hopper, Blackwell), AMD MI300x	FP8 paper, vLLM FP8
INT8 (W8A8)	8	channel / token	Turing onward in vLLM	vLLM
INT4 weight-only (AWQ, GPTQ)	4 + group scales	group, often 128	weights dequantized to 16-bit in the kernel	vLLM
MXFP4	4.25 effective	32 values, E8M0 power-of-two scale	Blackwell	NVIDIA
NVFP4	4.5 effective	16 values, FP8 E4M3 scale, plus one FP32 per tensor	Blackwell, Rubin	NVIDIA

FP8 comes in two flavours defined in the 2022 FP8 formats paper: E4M3 (more precision, used for weights and activations at inference) and E5M2 (more range, mostly for gradients). vLLM's docs put FP8 at a 2x memory reduction and up to 1.6x throughput with minimal accuracy impact, and older Turing and Ampere cards can still load FP8 checkpoints as weight-only W8A16 through Marlin kernels.

The 4-bit floats are the 2025 and 2026 story. Both MXFP4 and NVFP4 use the same E2M1 element, which can only represent 0, 0.5, 1, 1.5, 2, 3, 4 and 6 (and negatives). The difference is the scaling. MXFP4, from the OCP microscaling spec, shares a power-of-two scale across 32 values. NVFP4 shares a fractional FP8 scale across 16 values and adds a per-tensor FP32 scale, which NVIDIA says makes it about 3.5x smaller than FP16 and 1.8x smaller than FP8. MXFP4 got its public moment when OpenAI shipped gpt-oss with the MoE weights already in MXFP4, which is why gpt-oss-120b fits on a single 80 GB GPU despite 117B total parameters.

GGUF and k-quants: what Q4_K_M actually means

GGUF is a file format, not a quantization method. The spec describes a single self-contained file with typed key-value metadata (architecture, tokenizer, chat template) and tensor data aligned so it can be memory-mapped. Inside that container, each tensor can have its own type, and that is where the names come from.

• Legacy quants (Q4_0, Q5_1, Q8_0): simple blocks of 32 weights with a scale (and for _1, an offset). Q8_0 is the "nearly free" option.
• K-quants (Q2_K to Q6_K), introduced in PR #1684 in June 2023: super-blocks of 256 weights with sub-block scales that are themselves quantized. The suffix picks a mixture. Q4_K_M keeps some sensitive tensors, such as parts of ffn_down and attn_v, at 6-bit or higher, while Q4_K_S quantizes more aggressively.
• I-quants (IQ1_S to IQ4_XS): codebook-style types from about 1.56 to 4.25 bits per weight, designed to be used with an importance matrix.
• Float types: GGUF can now carry MXFP4 and NVFP4 tensors, and llama.cpp v0.9.0, released on 4 November 2026, added an OCP FP8 E4M3 type.

The 256-weight super-block has a practical consequence I hit while writing the companion step-by-step guide. SmolLM2-135M has a hidden size of 576, which is not divisible by 256, so llama-quantize (build b12110) fell back to q5_0 for 180 of 272 tensors and the "4-bit" file came out at 6.17 bits per weight. On 7B-and-up models with dimensions like 4096 this does not happen, but it is a reminder that the label on the file is a target, not a guarantee.

The other GGUF term you will see is imatrix, added in PR #4861 in January 2024. You run the full-precision model over some calibration text, record which weights see large activations, and pass that importance matrix to the quantizer so it spends its error budget where it hurts least. It is the same idea as AWQ, applied inside llama.cpp's own formats.

AWQ vs GPTQ vs round-to-nearest: how 4-bit weights are chosen

Round-to-nearest (RTN) is the baseline: pick a scale per group, round every weight. It is fine at 8 bits. At 4 bits the errors matter, and the two dominant GPU methods attack them differently.

GPTQ (Frantar et al., 2022) quantizes one column at a time and uses approximate second-order information to adjust the not-yet-quantized weights so the layer's output error stays small. The paper quantized a 175B model in about four GPU hours at 3 to 4 bits per weight.

AWQ (Lin et al., MLSys 2024 Best Paper) starts from the observation that protecting about 1% of salient weights greatly reduces error, and that you find those weights by looking at activations, not the weights themselves. Instead of keeping them in higher precision, it scales the salient channels up before quantization. There is no backpropagation, so it is less prone to overfitting the calibration set.

Both produce INT4 weight-only checkpoints (W4A16) that vLLM and TensorRT-LLM run through fast mixed-precision kernels, and both need a calibration set. In practice, who made the checkpoint and with what calibration data matters as much as the algorithm. MLX has its own versions too: mlx-lm ships AWQ, GPTQ, a dynamic per-layer mix and DWQ, which distils from the unquantized model.

How much quality does quantization lose?

Less than people fear at 8 bits, more than vendors imply below 4. The best public evidence comes from three places.

The llama.cpp perplexity README keeps a scoreboard for Llama 3 8B that measures KL divergence from the FP16 model's token distribution, which is a better quality signal than raw perplexity because it catches changes that happen to cancel out on average.

Read that chart as a cliff, not a slope. From Q8_0 down to Q4_K_M the curve is gentle: Q4_K_M agrees with the full model on the top token 91.9% of the time and loses 0.18 perplexity points on a base of 6.23. Below 4 bits it falls away fast, and Q2_K's KL divergence is fourteen times Q4_K_M's. The imatrix row shows the calibration effect is real but modest at 4 bits: 0.0282 against 0.0313. The README also notes that more Wikitext tokens in the imatrix did not consistently help.

On the GPU side, the largest public study I know is Kurtic, Alistarh and colleagues' "Give Me BF16 or Give Me Death"?, accepted at ACL 2025, which ran over 500,000 evaluations on the Llama 3.1 family. It found FP8 W8A8 effectively lossless at every scale, well-tuned INT8 W8A8 losing 1 to 3%, and INT4 weight-only more competitive than expected. Its deployment advice is about workload, not hardware: W4A16 is most cost-efficient for synchronous, low-batch serving, and W8A8 wins under asynchronous continuous batching.

For NVFP4, NVIDIA reports 1% or less degradation on DeepSeek-R1-0528 going from FP8 to NVFP4, for example 85% versus 84% on MMLU-Pro. That is a vendor number on a very large model, and large models tolerate quantization better than small ones. My own run on a 135M model in the guide saw Q4_K_M with imatrix reach a KL divergence of 0.035 and Q4_K_M without imatrix 0.048, both worse than Llama 3 8B's figures at the same nominal type.

Two caveats. Perplexity and KL divergence on Wikitext measure general text prediction; reasoning, code and tool calling can degrade more than the average suggests, so run your own eval on your own prompts. And numbers from different runtimes are not comparable: the README itself warns that llama.cpp perplexity values depend on implementation details.

Which quantization format for which runtime?

This is where most confusion comes from. A format is only useful if your runtime has a fast kernel for it on your hardware.

Runtime	Best-supported formats	Notes	Source
llama.cpp (and Ollama, LM Studio)	GGUF: Q4_K_M, Q5_K_M, Q6_K, Q8_0, IQ types, MXFP4	CPU, Metal, CUDA, Vulkan, ROCm; one file, easy to share	llama.cpp
vLLM	FP8 W8A8, INT4 AWQ/GPTQ via Marlin, INT8, compressed-tensors NVFP4, ModelOpt checkpoints	GGUF works but the docs call it "highly experimental and under-optimized"	vLLM, GGUF page
TensorRT-LLM	NVFP4, MXFP4, FP8 (per-tensor, block, rowwise), W4A16/W4A8 AWQ and GPTQ, FP8 and NVFP4 KV cache	NVFP4 and MXFP4 only on Blackwell and Rubin; Ampere gets W4A16 and FP8 KV cache	TensorRT-LLM
MLX (Apple silicon)	affine 1 to 8 bit (group 32, 64, 128), mxfp4, mxfp8, nvfp4	default is 4-bit affine with group size 64	MLX

In concrete terms, this is what the common paths look like. The llama.cpp commands are the ones I ran for the guide; the vLLM line is the example from vLLM's own GGUF page.

# llama.cpp b12110 (v0.9.0): BF16 GGUF -> importance matrix -> Q4_K_M
llama-imatrix  -m model-bf16.gguf -f calib.txt -o imatrix.gguf -c 512 --chunks 200
llama-quantize --imatrix imatrix.gguf model-bf16.gguf model-Q4_K_M.gguf Q4_K_M

# vLLM: GGUF is supported but experimental; use the base model's tokenizer
vllm serve unsloth/Qwen3-0.6B-GGUF:Q4_K_M --tokenizer Qwen/Qwen3-0.6B

My rule of thumb by hardware:

• Laptop, Mac or a single consumer GPU, one user: GGUF Q4_K_M with an imatrix as the default, Q5_K_M or Q6_K if you have the memory, Q8_0 if quality is the point. On a Mac, MLX 4-bit is the other strong option. My comparison of Ollama, llama.cpp, LM Studio and MLX covers the runtime choice.
• Hopper or Ada in a serving cluster: FP8 W8A8 first. It is close to lossless, needs no calibration in the dynamic variant, and both vLLM and TensorRT-LLM support it on these GPUs. Add FP8 KV cache when context length is the constraint. Drop to AWQ or GPTQ INT4 when the model does not fit.
• Blackwell: NVFP4 is the reason you bought it. Use published NVFP4 checkpoints or ModelOpt, and keep an FP8 build around for A/B evals.
• Ampere (A100, A10, RTX 30): no FP8 or FP4 tensor cores, so INT4 weight-only (AWQ or GPTQ via Marlin) or INT8 is the practical path.

If you are choosing between engines rather than formats, vLLM vs SGLang vs TensorRT-LLM compares them, and the format choice usually follows.

FAQ

Is Q4_K_M or Q5_K_M better?

Q5_K_M is closer to the original: on Llama 3 8B its KL divergence is 0.0108 against Q4_K_M's 0.0313, for a file about 16% larger (5.33 versus 4.58 GiB). If the bigger file fits in memory with room for your context, take Q5_K_M. If it does not, Q4_K_M with an imatrix is the usual sweet spot.

Is FP8 better than INT8 for LLM inference?

On hardware with FP8 tensor cores (Ada, Hopper, Blackwell, MI300x), generally yes. The large Llama 3.1 study found FP8 W8A8 effectively lossless while well-tuned INT8 W8A8 lost 1 to 3%. On Ampere and older, INT8 is the one with native support.

What is the difference between NVFP4 and MXFP4?

Both store values as 4-bit E2M1 floats. MXFP4 shares a power-of-two scale across 32 values; NVFP4 shares a more precise FP8 scale across 16 values plus a per-tensor FP32 scale, which costs a quarter of a bit more per value and tracks outliers better. Both need Blackwell-class hardware for native speed.

Does quantization make a model faster or just smaller?

Both, in different regimes. Weight-only quantization speeds up single-stream decode because it is bandwidth bound and there are fewer bytes to read. Weight-and-activation formats like FP8 W8A8 and NVFP4 also speed up compute-bound, heavily batched serving, but only on GPUs with matching tensor cores.

This quarter I would do three things: standardise on FP8 for anything served on Hopper, add a KL-divergence or task eval to the pipeline that produces every quantized artifact so a bad conversion fails CI instead of production, and keep GGUF Q4_K_M plus Q8_0 builds of each model for local and edge use. The formats will keep changing; the habit of measuring the quantized model against the original is what lasts. If you want to make your own GGUF files and measure them, the step-by-step llama.cpp quantization guide walks through it end to end.

Related: LLM Inference Engines 2026: vLLM vs SGLang vs TensorRT-LLM vs Dynamo, Self-Hosting LLMs vs API Costs in 2026: The 62% Utilisation Break-Even and Running LLMs Locally in 2026: Ollama vs llama.cpp vs LM Studio vs MLX.