How to Fine-Tune an LLM with LoRA Using Unsloth or TRL (2026)

This guide is for engineers who have decided a fine-tune is justified (if you have not, read when fine-tuning beats prompting and RAG first) and want a repeatable LoRA pipeline for an open-weight model rather than a notebook that worked once. At the end you will have a validated chat dataset, a LoRA adapter trained with either Unsloth or plain Hugging Face TRL, a side-by-side evaluation against the base model, and the same weights exported for vLLM (as a hot-loaded adapter) and for llama.cpp or Ollama (as a merged GGUF). Versions are current as of 20 October 2026: unsloth==2026.10.7 (which caps trl<=1.15.0 and transformers<=5.17.0), trl==1.16.0 with transformers==5.20.2 and peft==0.21.2 for the TRL path, vllm==0.32.0, and llama.cpp built from source. You need an NVIDIA GPU with compute capability 7.0 or newer, a CUDA 12.4+ driver (12.8+ for Blackwell), Python 3.11 to 3.13, and a Hugging Face account. I verified commands and arguments against the current docs and package metadata; the dataset validator output below is from a real run, but I have not reproduced a full training run for this post, so treat timings and loss values as yours to measure.

Step 1: Pick a base model, method and GPU

Pick the smallest model that can plausibly do the job; you can move up later with the same data. The examples use Qwen/Qwen3-4B-Instruct-2507 (Apache-2.0) because it trains comfortably on a 16 GB card. Start from an instruct model unless you are teaching a chat format from scratch.

Then choose the method by memory. Unsloth's requirements page publishes these minimums:

Model size	QLoRA (4-bit base)	LoRA (16-bit base)	Typical card
3B	3.5 GB	8 GB	T4 16 GB, RTX 4060 Ti
8B	6 GB	22 GB	L4 24 GB, RTX 4090
14B	8.5 GB	33 GB	L40S 48 GB
32B	26 GB	76 GB	A100 80 GB, H100
70B	41 GB	164 GB	H100 80 GB (QLoRA)

These are floors for short sequences. Real usage grows with max_seq_length and batch size, so leave 30 to 40 percent headroom. QLoRA is the default for a first run; LoRA Without Regret found LoRA matches full fine-tuning on small-to-medium datasets when applied to all layers, so you rarely need more than that.

Step 2: Create an isolated environment

Unsloth pins its own ranges of trl and transformers, so give each path its own virtual environment. Mixing them is the most common cause of import errors.

# Unsloth path (pulls compatible torch, trl<=1.15.0, transformers<=5.17.0)
uv venv .venv-unsloth --python 3.12
source .venv-unsloth/bin/activate
uv pip install "unsloth==2026.10.7" "vllm==0.32.0"
python -c "import unsloth, trl, transformers; print(unsloth.__version__, trl.__version__, transformers.__version__)"
deactivate

# Plain TRL path
uv venv .venv-trl --python 3.12
source .venv-trl/bin/activate
uv pip install "trl==1.16.0" "transformers==5.20.2" "peft==0.21.2" \
               "bitsandbytes==0.50.2" "accelerate==1.15.0" "datasets==5.1.0"
nvidia-smi --query-gpu=name,memory.total,compute_cap --format=csv

If vllm and unsloth resolve to conflicting torch builds on your platform, put vLLM in a third environment; it only needs the adapter directory from Step 4.

Step 3: Prepare and validate the dataset

Use the conversational format both TRL and Unsloth understand: one JSON object per line, a messages list, and the assistant's ideal reply as the last turn. Put the same system prompt you will use in production on every row.

{"messages": [{"role": "system", "content": "Classify the support ticket. Reply with JSON only."}, {"role": "user", "content": "My invoice shows two charges for October."}, {"role": "assistant", "content": "{\"category\": \"billing\", \"priority\": \"high\"}"}]}

Quality beats volume. A few hundred consistent, correct examples beat ten thousand scraped ones, and duplicates or contradictory labels show up later as a loss curve that will not settle. Run a validator before every training job:

# validate_dataset.py: sanity-check a chat-format JSONL file before training.
# Usage: python validate_dataset.py data/train.jsonl
import json, sys

path = sys.argv[1]
roles_ok = {"system", "user", "assistant"}
errors, seen, n, lengths = [], set(), 0, []

with open(path, encoding="utf-8") as f:
    for i, line in enumerate(f, 1):
        n += 1
        try:
            row = json.loads(line)
        except json.JSONDecodeError as e:
            errors.append(f"line {i}: invalid JSON ({e.msg})"); continue
        msgs = row.get("messages")
        if not isinstance(msgs, list) or not msgs:
            errors.append(f"line {i}: missing 'messages' list"); continue
        if any(m.get("role") not in roles_ok or not str(m.get("content", "")).strip() for m in msgs):
            errors.append(f"line {i}: bad role or empty content")
        if msgs[-1].get("role") != "assistant":
            errors.append(f"line {i}: last message is not from the assistant")
        key = json.dumps(msgs, sort_keys=True)
        if key in seen:
            errors.append(f"line {i}: exact duplicate")
        seen.add(key)
        lengths.append(sum(len(m.get("content", "")) for m in msgs))

lengths.sort()
print(f"rows: {n}  errors: {len(errors)}")
if lengths:
    print(f"chars per row: median {lengths[len(lengths)//2]}, max {lengths[-1]}")
for e in errors[:20]:
    print("  " + e)
sys.exit(1 if errors else 0)

On a deliberately broken four-row file it prints:

rows: 4  errors: 2
chars per row: median 123, max 134
  line 3: exact duplicate
  line 4: last message is not from the assistant

Then hold out an evaluation set before you look at any results, and never train on it:

python - <<'EOF'
import random
rows = open("data/all.jsonl", encoding="utf-8").read().splitlines()
random.Random(3407).shuffle(rows)
cut = max(50, len(rows) // 10)          # at least 50 rows, or 10 percent
open("data/eval.jsonl", "w").write("\n".join(rows[:cut]) + "\n")
open("data/train.jsonl", "w").write("\n".join(rows[cut:]) + "\n")
print(f"train={len(rows) - cut} eval={cut}")
EOF
python validate_dataset.py data/train.jsonl && python validate_dataset.py data/eval.jsonl

Step 4: Train a LoRA adapter with Unsloth

This follows the structure of Unsloth's own Qwen3 notebook with the hyperparameters from its LoRA hyperparameters guide: rank 16 to 32, alpha equal to rank, LoRA on all attention and MLP projections, learning rate 2e-4, one to three epochs.

# train_unsloth.py  (unsloth==2026.10.7)
from unsloth import FastLanguageModel          # import unsloth before trl/transformers
from unsloth.chat_templates import get_chat_template, train_on_responses_only
from datasets import load_dataset
from trl import SFTTrainer, SFTConfig

MAX_LEN = 2048

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Qwen3-4B-Instruct-2507",
    max_seq_length=MAX_LEN,
    load_in_4bit=True,            # QLoRA; set False and load_in_16bit=True for 16-bit LoRA
)

model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    lora_alpha=16,
    lora_dropout=0,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    bias="none",
    use_gradient_checkpointing="unsloth",   # trades compute for memory on long sequences
    random_state=3407,
)

tokenizer = get_chat_template(tokenizer, chat_template="qwen3-instruct")

def to_text(batch):
    return {"text": [tokenizer.apply_chat_template(m, tokenize=False, add_generation_prompt=False)
                     for m in batch["messages"]]}

ds = load_dataset("json", data_files={"train": "data/train.jsonl", "eval": "data/eval.jsonl"})
ds = ds.map(to_text, batched=True)

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,                 # Unsloth's patched trainer accepts tokenizer=
    train_dataset=ds["train"],
    eval_dataset=ds["eval"],
    args=SFTConfig(
        output_dir="outputs/unsloth",
        dataset_text_field="text",
        max_length=MAX_LEN,
        per_device_train_batch_size=2,
        gradient_accumulation_steps=8,   # effective batch size 16
        num_train_epochs=2,
        learning_rate=2e-4,
        warmup_steps=0.05,               # float below 1 = ratio of total steps
        lr_scheduler_type="linear",
        weight_decay=0.01,
        optim="adamw_8bit",
        logging_steps=10,
        eval_strategy="steps",
        eval_steps=50,
        save_steps=50,
        save_total_limit=3,
        seed=3407,
        report_to="none",
    ),
)

# Compute loss on assistant replies only, not on the system/user text.
trainer = train_on_responses_only(
    trainer,
    instruction_part="<|im_start|>user\n",
    response_part="<|im_start|>assistant\n",
)

trainer.train()
model.save_pretrained("adapters/ticket-lora")      # LoRA weights only
tokenizer.save_pretrained("adapters/ticket-lora")

source .venv-unsloth/bin/activate
python train_unsloth.py 2>&1 | tee train.log

Before the long run, set max_steps=30 and decode one row of trainer.train_dataset to confirm the labels only cover the assistant text. Unsloth reports roughly a one-point accuracy gain from response-only training, more on multi-turn data.

Step 5: Or train the same adapter with plain TRL and PEFT

If you need multi-GPU training, a model Unsloth does not support yet, or fewer moving parts, TRL does the same job. Convert each row to a conversational prompt-completion pair; TRL then computes loss on the completion only by default (completion_only_loss is effectively on for prompt-completion data, per the SFTTrainer docs).

# train_trl.py  (trl==1.16.0, transformers==5.20.2, peft==0.21.2, bitsandbytes==0.50.2)
import torch
from datasets import load_dataset
from peft import LoraConfig
from transformers import BitsAndBytesConfig
from trl import SFTConfig, SFTTrainer

def split_last_turn(row):
    return {"prompt": row["messages"][:-1], "completion": row["messages"][-1:]}

ds = load_dataset("json", data_files={"train": "data/train.jsonl", "eval": "data/eval.jsonl"})
ds = ds.map(split_last_turn, remove_columns=["messages"])

trainer = SFTTrainer(
    model="Qwen/Qwen3-4B-Instruct-2507",
    train_dataset=ds["train"],
    eval_dataset=ds["eval"],
    quantization_config=BitsAndBytesConfig(          # QLoRA: 4-bit NF4 base weights
        load_in_4bit=True,
        bnb_4bit_quant_type="nf4",
        bnb_4bit_use_double_quant=True,
        bnb_4bit_compute_dtype=torch.bfloat16,
    ),
    peft_config=LoraConfig(
        r=16, lora_alpha=16, lora_dropout=0.0,
        target_modules="all-linear", task_type="CAUSAL_LM",
    ),
    args=SFTConfig(
        output_dir="outputs/trl",
        model_init_kwargs={"dtype": torch.bfloat16},  # SFTTrainer defaults to float32 otherwise
        max_length=2048,                              # TRL default is 1024
        per_device_train_batch_size=2,
        gradient_accumulation_steps=8,
        num_train_epochs=2,
        learning_rate=2e-4,                           # TRL default 2e-5 is for full fine-tuning
        warmup_steps=0.05,
        lr_scheduler_type="cosine",
        logging_steps=10,
        eval_strategy="steps",
        eval_steps=50,
        save_steps=50,
        save_total_limit=3,
        report_to="none",
    ),
)
trainer.train()
trainer.save_model("adapters/ticket-lora-trl")

source .venv-trl/bin/activate
accelerate launch train_trl.py      # or: python train_trl.py on a single GPU

Step 6: Read the loss curve before you trust it

Watch loss and eval_loss together. Unsloth's guide says a training loss around 0.5 to 1.0 is often healthy and below about 0.2 usually means memorisation; on short, highly structured outputs like the JSON above, loss can legitimately sit lower, which is why the held-out eval in Step 7 matters more than any single number.

• Training loss falls, eval loss rises: overfitting. Cut epochs, lower the learning rate, add lora_dropout=0.1, or add data.
• Both flat from step 1: learning rate too low, or every label is masked (check the decoded labels).
• Spikes to NaN: learning rate too high or fp16 overflow; switch to bf16 if the GPU supports it.

Step 7: Evaluate the adapter against the base model

Run the base model and the adapter over the held-out set with identical prompts and sampling, and score with a check that reflects the real task. vLLM can serve both from one engine:

# eval_adapter.py  (vllm==0.32.0)
import json
from vllm import LLM, SamplingParams
from vllm.lora.request import LoRARequest

rows = [json.loads(l) for l in open("data/eval.jsonl", encoding="utf-8")]
prompts = [r["messages"][:-1] for r in rows]
expected = [json.loads(r["messages"][-1]["content"]) for r in rows]

llm = LLM(model="Qwen/Qwen3-4B-Instruct-2507", enable_lora=True, max_lora_rank=16)
params = SamplingParams(temperature=0.0, max_tokens=128)

def score(outputs):
    ok = 0
    for out, exp in zip(outputs, expected):
        try:
            ok += json.loads(out.outputs[0].text) == exp     # exact structured match
        except json.JSONDecodeError:
            pass
    return ok / len(expected)

base = llm.chat(prompts, params)
tuned = llm.chat(prompts, params, lora_request=LoRARequest("ticket", 1, "adapters/ticket-lora"))
print(f"base exact-match: {score(base):.1%}   adapter exact-match: {score(tuned):.1%}")

Also run a fixed set of general prompts (refusals, plain instructions, other formats your product relies on) through both, since LoRA forgets less than full fine-tuning but not nothing. Put this script in CI so a retrain cannot regress silently; the LLM evals in GitHub Actions guide covers the wiring.

Step 8: Export to vLLM, GGUF and Ollama

Serve the adapter on vLLM without merging. This keeps one copy of the base weights on the GPU and lets you hot-swap adapters (the serving trade-offs are in LLM inference in 2026). vLLM 0.32.0's default max_lora_rank is 16, so pass the rank you trained with:

vllm serve Qwen/Qwen3-4B-Instruct-2507 \
  --enable-lora \
  --lora-modules ticket=./adapters/ticket-lora \
  --max-lora-rank 16 \
  --max-loras 4

# Select the adapter by name in the "model" field
curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "ticket",
  "messages": [{"role": "system", "content": "Classify the support ticket. Reply with JSON only."},
               {"role": "user", "content": "I was charged twice."}],
  "temperature": 0
}'

For the Kubernetes version of this, see deploying vLLM on Kubernetes with GPUs.

Merge and write GGUF with Unsloth. From the Unsloth environment, after loading the adapter with FastLanguageModel.from_pretrained(model_name="adapters/ticket-lora", max_seq_length=2048, load_in_4bit=True):

# Merged 16-bit safetensors, loadable by vLLM or transformers without PEFT
model.save_pretrained_merged("merged/ticket-16bit", tokenizer, save_method="merged_16bit")

# GGUF for llama.cpp / Ollama; q4_k_m is the usual size/quality trade-off
model.save_pretrained_gguf("gguf/ticket", tokenizer, quantization_method="q4_k_m")

Merge and convert on the TRL path. Merge with PEFT, then use llama.cpp's converter and quantiser:

python - <<'EOF'
import torch
from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer
m = AutoPeftModelForCausalLM.from_pretrained("adapters/ticket-lora-trl", dtype=torch.bfloat16)
m.merge_and_unload().save_pretrained("merged/ticket-16bit")
AutoTokenizer.from_pretrained("adapters/ticket-lora-trl").save_pretrained("merged/ticket-16bit")
EOF

git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build -DGGML_CUDA=ON
cmake --build llama.cpp/build --config Release -j --target llama-quantize llama-server
pip install -r llama.cpp/requirements.txt
python llama.cpp/convert_hf_to_gguf.py merged/ticket-16bit --outfile ticket-bf16.gguf --outtype bf16
./llama.cpp/build/bin/llama-quantize ticket-bf16.gguf ticket-Q4_K_M.gguf Q4_K_M

Run it in Ollama. Point a Modelfile at the GGUF and set parameters; afterwards, check with ollama show ticket --modelfile that the chat template matches the one you trained with, and add a TEMPLATE block if it does not:

# Modelfile
FROM ./ticket-Q4_K_M.gguf
PARAMETER temperature 0
PARAMETER stop "<|im_end|>"

ollama create ticket -f Modelfile
ollama run ticket "I was charged twice for October."

Rerun the Step 7 evaluation against the exported model. The Ollama guide covers serving it beyond your laptop.

Troubleshooting

Symptom	Likely cause	Fix
torch.OutOfMemoryError: CUDA out of memory at step 0	Model plus activations exceed VRAM	Use QLoRA (load_in_4bit=True), batch size 1 with higher gradient_accumulation_steps, lower max_length, keep gradient checkpointing on
OOM only on some steps, mid-epoch	A few very long rows	Check max length in the validator; drop or split outliers; set max_length explicitly
OOM while saving GGUF or merged 16-bit	Merge needs a full-precision copy	Unsloth docs: pass maximum_memory_usage=0.5 to the save call; or merge on a larger GPU or CPU
Loss flat from the first step	Labels fully masked, or learning rate too low for LoRA	Decode labels for one row; check response_part matches the template; use 2e-4, not TRL's 2e-5 default
Training loss near 0, eval loss rising	Overfitting or duplicated data	Fewer epochs, lora_dropout=0.1, dedupe, add examples, keep the earliest good checkpoint
Loss becomes nan	fp16 overflow or learning rate too high	Use bf16 on Ampere or newer; halve the learning rate; keep max_grad_norm=1.0
pip resolver conflict on trl or transformers	unsloth==2026.10.7 caps trl<=1.15.0, transformers<=5.17.0	Separate virtual environments for Unsloth and plain TRL
Unsloth warns it was imported after transformers	Patches not applied	Put from unsloth import FastLanguageModel first in the script
vLLM rejects adapter: rank larger than max	vLLM default max_lora_rank is 16	Pass --max-lora-rank equal to your adapter's r (allowed values include 8, 16, 32, 64)
Good in training, gibberish or endless output in llama.cpp or Ollama	Chat template, EOS or BOS mismatch	Use the training template; set the stop token (`<

FAQ

How much GPU memory do I need to fine-tune an 8B model?

With QLoRA, Unsloth lists a 6 GB minimum for 8B, and a 24 GB card such as an L4 or RTX 4090 is comfortable with 2,048-token sequences. With 16-bit LoRA the floor is 22 GB, so plan on 40 to 48 GB. Sequence length and batch size move these numbers more than anything else.

Should I use Unsloth or TRL?

Use Unsloth on a single GPU when it supports your model: it trains faster with less memory and has built-in GGUF export. Use plain TRL with PEFT for multi-GPU jobs through accelerate, very new architectures, or when you want the fewest dependencies. Both produce standard PEFT adapters that vLLM loads.

What LoRA rank and learning rate should I use?

Start at r=16, lora_alpha=16, all linear layers, learning rate 2e-4, one to three epochs. Raise the rank to 32 or 64 only if the model underfits a large dataset. Thinking Machines found the best LoRA learning rate is roughly 10 times the full fine-tuning rate, which is why TRL's 2e-5 default is too low for adapters.

Should I merge the adapter or keep it separate?

Keep it separate if you serve on vLLM and might run several adapters on one base model, or want to swap versions quickly. Merge when the target needs a single file, such as GGUF for llama.cpp or Ollama, or when you want to avoid adapter overhead at inference time.

When a fine-tune misbehaves, I check three things before touching hyperparameters: the decoded labels for one training row (is the loss on the right tokens?), the validator output (duplicates and wrong last turns), and the chat template in the serving stack. Most failures I have seen trace back to one of those, not to the learning rate. In production I keep the adapter, its training data hash, the eval score and the base model revision together as one release artefact, so a rollback is a flag change and a retrain on the next base model is a rerun. Official docs: Unsloth, TRL SFTTrainer, vLLM LoRA adapters and llama.cpp quantize.

Related: MLOps in 2026: Evals Are the New Unit Tests, Judges Agree With Experts 66% of the Time, and OpenAI's Evals API Shuts Down on 30 November, LLM Inference in 2026: Prefill Is a Different Machine Now, and the KV Cache Has a Disk Tier and On-Device Inference in 2026: 160 Tokens a Second on a Laptop, a 2-Bit Model in Every iPhone, and the Bandwidth Arithmetic That Decides What Stays Local.