How to Quantize an LLM to GGUF with llama.cpp: Q4_K_M and imatrix (2026)

This guide is for engineers who want to make their own GGUF files instead of trusting whichever Q4 upload appeared first on Hugging Face. By the end you will have a BF16 GGUF, an importance matrix, Q4_K_M, Q5_K_M and Q8_0 quantizations, a KL-divergence report that tells you how far each one drifted from the original, and an OpenAI-compatible endpoint serving the winner. I ran every command below on 10 November 2026 with llama.cpp build b12110 (the build behind the v0.9.0 release), Python 3.12, PyTorch 2.11.0 and transformers 4.57.6, using SmolLM2-135M-Instruct so the whole pipeline finishes in a couple of minutes on a laptop. You need about 3 GB of free disk, Python 3.10 or newer, and enough RAM to hold the model in 16-bit (roughly two bytes per parameter). If you want the background on what the formats mean first, read LLM quantization explained.

Step 1: Get llama.cpp b12110 and the converter's Python dependencies

You need two halves of llama.cpp: the compiled tools (llama-quantize, llama-imatrix, llama-perplexity, llama-server) and the Python conversion script from the same commit. Pin both to the same build. The converter and the C++ loader evolve together, and a GGUF written by a newer script can fail on an older binary.

The prebuilt binaries are the fastest route. Each build on the releases page ships tarballs for macOS, Ubuntu CPU, CUDA 12.8, CUDA 13.4, Vulkan, ROCm and more.

# Pick the tarball for your platform; macOS Apple Silicon shown.
# Linux + NVIDIA: llama-b12110-bin-ubuntu-cuda-12.8-x64.tar.gz (or cuda-13.4)
export LLAMA_BUILD=b12110
mkdir -p ~/quant && cd ~/quant
curl -L -o llama.tgz \
  https://github.com/ggml-org/llama.cpp/releases/download/${LLAMA_BUILD}/llama-${LLAMA_BUILD}-bin-macos-arm64.tar.gz
tar -xzf llama.tgz                     # extracts to ./llama-b12110/
export PATH="$PWD/llama-${LLAMA_BUILD}:$PATH"
llama-cli --version

Expected output:

version: 0.9.0-dev (build 12110, commit cd959bae9)
built with AppleClang 21.0.0.21000101 for Darwin arm64

If you would rather compile, the build docs use plain CMake; add -DGGML_CUDA=ON for NVIDIA GPUs.

# Optional: build from source instead (CUDA example)
git clone --depth 1 --branch b12110 https://github.com/ggml-org/llama.cpp llama.cpp-src
cmake -S llama.cpp-src -B llama.cpp-src/build -DGGML_CUDA=ON
cmake --build llama.cpp-src/build --config Release -j 8

Now the Python side. Clone the same tag (shallow is fine) and install the pinned requirements into a virtual environment. At b12110 the converter pins torch==2.11.0 from the CPU wheel index and transformers==4.57.6.

cd ~/quant
git clone --depth 1 --branch b12110 https://github.com/ggml-org/llama.cpp llama.cpp-src
python3 -m venv .venv && source .venv/bin/activate
pip install -r llama.cpp-src/requirements/requirements-convert_hf_to_gguf.txt
python -c "import torch, transformers; print(torch.__version__, transformers.__version__)"
# 2.11.0 4.57.6

Step 2: Download the model from Hugging Face

The hf command comes with huggingface_hub, which transformers already installs. Download only what the converter needs: weights, configs and tokenizer files. Many repos also carry ONNX exports, GGUFs or training logs that you do not want; SmolLM2's repo pulled an onnx/ folder when I forgot the filter.

# Gated models (Llama, Gemma): run `hf auth login` first, or export HF_TOKEN
hf download HuggingFaceTB/SmolLM2-135M-Instruct \
  --include "*.safetensors" "*.json" "*.txt" "tokenizer.model" \
  --local-dir models/SmolLM2-135M-Instruct
ls models/SmolLM2-135M-Instruct
# config.json  generation_config.json  merges.txt  model.safetensors
# special_tokens_map.json  tokenizer.json  tokenizer_config.json  vocab.json ...

Check config.json for the architecture and native dtype before you go further. SmolLM2 reports "torch_dtype": "bfloat16" and a hidden_size of 576, a detail that matters in Step 5.

Step 3: Convert safetensors to a BF16 GGUF

convert_hf_to_gguf.py reads the Hugging Face checkpoint and writes a single GGUF with tensors, tokenizer and chat template embedded. Its --outtype accepts f32, f16, bf16, q8_0, tq1_0, tq2_0 and auto (the default), which keeps supported FP8 weights and otherwise picks the highest-fidelity 16-bit type. I set bf16 explicitly so the baseline matches the model's training dtype and the file name is predictable.

python llama.cpp-src/convert_hf_to_gguf.py models/SmolLM2-135M-Instruct \
  --outtype bf16 \
  --outfile models/smollm2-135m-bf16.gguf

The tail of the output should look like this (abridged):

INFO:gguf.gguf_writer:models/smollm2-135m-bf16.gguf: n_tensors = 272, total_size = 269.1M
INFO:hf-to-gguf:Model successfully exported to models/smollm2-135m-bf16.gguf

Run the script from the cloned repo as shown; it imports a conversion package that sits next to it. For big models, --split-max-size 50G writes shards and --use-temp-file lowers peak RAM. The converter cannot write k-quants, and llama-quantize refuses to requantize an already-quantized file by default, so the BF16 GGUF is your master copy. Keep it.

Step 4: Build an importance matrix with llama-imatrix

An importance matrix records which weights see large activations on real text, so the quantizer can protect them. The imatrix README needs only a model and a calibration file; output defaults to GGUF format. I used the first 2 MB of Wikitext-2's training split, which the repo's scripts/get-wikitext-2.sh downloads.

mkdir -p data && (cd data && sh ../llama.cpp-src/scripts/get-wikitext-2.sh)
head -c 2000000 data/wikitext-2-raw/wiki.train.raw > data/calib.txt

llama-imatrix -m models/smollm2-135m-bf16.gguf -f data/calib.txt \
  -o models/imatrix.gguf -c 512 --chunks 200 -ngl 99

compute_imatrix: computing over 200 chunks, n_ctx=512, batch_size=2048, n_seq=4
...
Final estimate: PPL = 20.4101 +/- 0.27312

The result was a 1.3 MB imatrix.gguf in under a minute. For a real deployment, calibrate on text that looks like your traffic: code for a coding model, chat transcripts for an assistant, your language if it is not English. Do not calibrate on the same text you will evaluate on, or your perplexity numbers will flatter you. -ngl 99 offloads all layers to the GPU (Metal here); the flag also accepts all or auto.

Step 5: Quantize to Q4_K_M, Q5_K_M and Q8_0 with llama-quantize

The syntax is llama-quantize [options] input.gguf output.gguf TYPE. Run llama-quantize --help to see every type with its reference size and perplexity cost on Llama 3 8B; Q4_K_M is listed at 4.58 GB and +0.1754 perplexity there. Use --dry-run to preview the size without writing anything.

# 8-bit: no imatrix needed, near-lossless
llama-quantize models/smollm2-135m-bf16.gguf models/smollm2-135m-Q8_0.gguf Q8_0

# k-quants with the importance matrix
llama-quantize --imatrix models/imatrix.gguf \
  models/smollm2-135m-bf16.gguf models/smollm2-135m-Q5_K_M.gguf Q5_K_M
llama-quantize --imatrix models/imatrix.gguf \
  models/smollm2-135m-bf16.gguf models/smollm2-135m-Q4_K_M.gguf Q4_K_M

# Control: Q4_K_M without imatrix, to see what calibration buys
llama-quantize models/smollm2-135m-bf16.gguf models/smollm2-135m-Q4_K_M-noimat.gguf Q4_K_M

The Q4_K_M log ends with something you should read rather than skip:

load_imatrix: loaded 210 importance matrix entries from models/imatrix.gguf computed on 200 chunks
warning: blk.0.attn_k.weight - ncols 576 not divisible by 256 (required for type q4_K) -> falling back to q5_0
...
llama_model_quantize_impl: model size  =   256.63 MiB (16.00 BPW)
llama_model_quantize_impl: quant size  =    98.87 MiB (6.17 BPW)
llama_model_quantize_impl: WARNING: 180 of 272 tensor(s) required fallback quantization

K-quants work on 256-weight super-blocks, and a 576-wide tensor does not divide evenly, so llama.cpp falls back to q5_0 for those tensors. That is why this "4-bit" file is 6.17 bits per weight. Models with dimensions like 4096 or 5120 do not hit this. With tiny or oddly shaped models, compare bits per weight, not type names.

Step 6: Measure perplexity and KL divergence with llama-perplexity

Perplexity alone hides a lot, so use the KL-divergence mode described in the perplexity README. First run the BF16 model once to save its logits, then score each quantization against that file. Evaluate on the test split, not the text you calibrated on.

T=data/wikitext-2-raw/wiki.test.raw

# 1. Baseline: save BF16 logits (big file; see note below)
llama-perplexity -m models/smollm2-135m-bf16.gguf -f $T -c 512 --chunks 100 -ngl 99 \
  --kl-divergence-base models/base.kld

# 2. Each quant against the baseline (the text is read from base.kld)
for q in Q8_0 Q5_K_M Q4_K_M Q4_K_M-noimat; do
  echo "== $q"
  llama-perplexity -m models/smollm2-135m-$q.gguf -c 512 --chunks 100 -ngl 99 \
    --kl-divergence-base models/base.kld --kl-divergence 2>&1 \
    | grep -E "Mean PPL\(Q\)|Mean    KLD|Same top p"
done

Results from my run, 100 chunks of 512 tokens on an Apple Silicon Mac with Metal:

File	Size (MiB)	Bits/weight	PPL	Mean KLD	Same top token
BF16 (baseline)	256.63	16.00	18.90	0	100%
Q8_0	136.40	8.51	18.94	0.0014	97.5%
Q5_K_M + imatrix	105.21	6.56	19.05	0.0121	93.7%
Q4_K_M + imatrix	98.87	6.17	19.53	0.0347	89.8%
Q4_K_M, no imatrix	98.87	6.17	19.78	0.0484	87.4%

Two things stand out. The imatrix cut KL divergence at Q4_K_M by about 28% (0.0484 to 0.0347) for an identical file size, so it is worth the minute it takes. And Q8_0 is close to indistinguishable from BF16 at half the size. For comparison, the README's Llama 3 8B scoreboard puts Q4_K_M with imatrix at a KLD of 0.0282: bigger models tolerate quantization better. Note that the baseline logits file was 2.5 GB for just 100 chunks of this 49,152-token vocabulary; the README warns it reaches 37 GiB for Llama 3 on the full test set. Delete it when you are done.

Step 7: Serve the GGUF with llama-server

llama-server exposes an OpenAI-compatible API and reads the chat template from the GGUF metadata, with Jinja templating on by default in this build.

llama-server -m models/smollm2-135m-Q4_K_M.gguf -c 4096 -ngl 99 \
  --host 127.0.0.1 --port 8080
# in another shell
curl -s http://127.0.0.1:8080/health
curl -s http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"What is the capital of France? One word."}],"max_tokens":16,"temperature":0}'

Expected output, abridged (the token counts match my server log; the reply text itself is illustrative and will vary):

{"status":"ok"}
{"choices":[{"finish_reason":"stop","index":0,
  "message":{"role":"assistant","content":"The capital of France is Paris."}}],
 "system_fingerprint":"b12110-cd959bae9", "usage":{"completion_tokens":8,"prompt_tokens":40,"total_tokens":48}}

It ignored "one word", which is a 135M-parameter model being a 135M-parameter model, not a quantization artefact. Bind to 127.0.0.1 unless you also set --api-key. If you want to benchmark it properly, How to benchmark LLM inference covers TTFT and throughput, and the same GGUF loads in Ollama through a Modelfile FROM line; see the Ollama guide.

Troubleshooting

Symptom	Likely cause	Fix
BPE pre-tokenizer was not recognized - update get_vocab_base_pre()	Converter older than the model's tokenizer	Use a newer llama.cpp build for both converter and binaries; do not hand-edit the hash
NotImplementedError: Architecture '...' not supported!	Model architecture not in this build	Check --print-supported-models; upgrade to a build that lists it
requantizing from type q8_0 is disabled	Quantizing an already-quantized GGUF	Quantize from the BF16 or F16 file; --allow-requantize works but loses quality
ncols 576 not divisible by 256 ... falling back to q5_0	Tensor width not a multiple of the 256-weight k-quant super-block	Expected on small models; compare bits per weight, or use Q4_0/Q5_0 deliberately
Converter killed or out of memory	Whole model held in RAM	Add --use-temp-file; make sure RAM covers the 16-bit size; shard with --split-max-size
401 or gated-repo error from hf download	No accepted licence or token	Accept the licence on the model page, then hf auth login or set HF_TOKEN
KLD baseline fills the disk	Logits saved for every token and vocab entry	Limit with --chunks; delete base.kld afterwards
Very slow imatrix or perplexity runs	Layers left on CPU, or a CPU-only tarball	Use the CUDA, ROCm or Vulkan build and pass -ngl 99
Chat replies ignore roles or ramble	Missing or wrong chat template in metadata	Check tokenizer_config.json has chat_template; override with --chat-template

FAQ

Which GGUF quantization should I use?

Start with Q4_K_M plus an imatrix if memory is tight, Q5_K_M or Q6_K if it is not, and Q8_0 when you want results indistinguishable from the original. Then measure KL divergence on your own text as in Step 6 rather than trusting the type name.

Do I need an imatrix for Q8_0 or Q6_K?

It is not required, and at 8 bits the gain is negligible. It starts to matter at 4 bits (a 28% KLD reduction in my run) and is effectively mandatory for IQ2 and IQ3 types.

Can I convert straight to Q4_K_M with convert_hf_to_gguf.py?

No. The converter only writes f32, f16, bf16, q8_0, the two ternary types, or auto. K-quants come from llama-quantize, which is also where the imatrix is applied.

How long does quantizing a 70B model take?

The conversion and quantization steps are mostly disk and memory bound; the imatrix step is the slow one because it runs inference over the calibration text, so it scales with model size and GPU. I have not timed a 70B run for this guide, so treat any number you read as specific to that hardware.

When I take a new model into production, the first thing I check is the bits-per-weight line and the fallback warning in the llama-quantize log, then the same-top-token percentage from Step 6 on a few thousand tokens of real prompts. Anything that loses more than a few points there gets bumped one size up. I also keep the BF16 GGUF and the imatrix as build artefacts next to the quantized files, because regenerating them is where most of the time goes. The official references are the quantize README, the imatrix README, the perplexity README and the GGUF spec.

Related: How to Run a Local LLM With Ollama: Setup, GPU and API Guide (2026), How to Benchmark LLM Inference: TTFT, Throughput and Goodput (2026) and How to Deploy vLLM on Kubernetes with GPUs: Step-by-Step (2026).