PEFT: LoRA & QLoRA
Full fine-tuning updates every weight of a 7B-parameter model — that's 14 GB just for the weights at FP16, plus another 14 GB of gradients and ~28 GB of Adam optimizer state (the two moments), pushing the total past 60 GB. The whole exercise often won't even fit on the GPU you have. Parameter-Efficient Fine-Tuning (PEFT) says: most of the model is already great, so freeze it. Only train a tiny new piece on top.
Real-world examples: every "fine-tune Llama on my support tickets" tutorial; Stable Diffusion LoRAs that teach a style or a face; a 65B model adapted on a single 48 GB GPU with QLoRA; Hugging Face's peft library powering most open-weight community adapters.
Full FT vs parameter-efficient FT
| Full fine-tuning | LoRA | QLoRA | |
|---|---|---|---|
| Trainable params | all (e.g. 7B) | a few million | a few million |
| Base model precision | FP16 / BF16 | FP16 / BF16 | 4-bit NF4 |
| GPU memory (7B) | ≈ 60 GB | ≈ 18 GB | ≈ 6 GB |
| Stores per task | full new model | small adapter file | small adapter file |
| Speed | slowest | fast | fast |
The cost gap is so large that PEFT is the default in 2025 — full fine-tuning is reserved for foundation labs and base-model rewrites.
The low-rank update
LoRA's claim is empirical but powerful: the weight update a fine-tune wants to make to a pretrained matrix has very low intrinsic rank. So instead of learning a dense update , learn two skinny matrices and multiply them:
The frozen base stays untouched; the forward pass becomes
At initialization is a random Gaussian and , so : the adapter starts as an exact no-op and the model behaves identically to the pretrained base, only departing from it as training pulls away from zero. This is what makes LoRA safe to attach without degrading the base model on step one.
At inference you can either keep and separate (zero added latency for an attached adapter) or merge them into for a vanilla single matrix multiply.
Low-rank update ΔW ≈ B · A
A full update ΔW (left) is approximated by two skinny matrices: B (d×r) and A (r×d). Sliding rank r changes how many parameters LoRA stores — and how close the reconstruction gets. The target here has intrinsic rank 4, so r = 4 reconstructs it perfectly.
Counting parameters makes the savings concrete. For a single attention projection:
- Full update: params.
- LoRA, rank 8: params.
That's a 256× reduction for one matrix; multiplied across all attention/MLP projections in a transformer, you go from "needs a node of A100s" to "fits on one consumer GPU".
Rank and alpha
LoRA has two knobs you actually tune.
- Rank
r: the bottleneck width. Common values: 4, 8, 16, 32, 64. Higher rank = more capacity, more memory, more risk of overfitting on a small dataset. - Alpha
α: a scaling factor. The effective update is , so changing doesn't silently change the update magnitude. Treat as a learning-rate-like knob; a common heuristic is , but many recipes pin and only sweep .
Where to attach adapters
In transformers, LoRA is most commonly applied to the attention query and value projections (q_proj, v_proj). Adding it to k_proj, o_proj, and the MLP up/down/gate projections gives a small extra bump but costs proportionally more parameters. Start with q_proj + v_proj; only widen if you're capacity-bound.
Adapters as a deployment story
Because a LoRA adapter is a tiny file (often 10–200 MB), you can:
- Hot-swap personas or skills: one base model in GPU memory, dozens of adapters on disk, load one per request.
- Ship updates without re-downloading the base.
- Compose: some serving stacks support multiple adapters merged with weights at inference time.
This is a structural advantage over full fine-tuning, which gives you a fresh multi-GB checkpoint every time.
QLoRA: the 4-bit twist
Even with LoRA, the frozen base still has to live in GPU memory. A 65B-parameter model at FP16 is 130 GB — nothing single-GPU fits it. QLoRA adds three ingredients on top of LoRA:
- 4-bit NF4 quantization of the frozen base. "NF4" (NormalFloat-4) is a quantization scheme designed for the bell-shaped weight distributions you actually see in language models. It has 16 levels placed at the quantiles of a unit normal, not evenly spaced.
- Double quantization: the per-block quantization constants are themselves quantized, saving an extra ~0.4 bits per parameter.
- Paged optimizer state: Adam state spills to CPU memory under pressure instead of crashing the run.
Critically, the LoRA adapters stay in higher precision (BF16). Quantization compresses the frozen part; the trainable part is small enough that precision doesn't hurt.
The result: a 65B model becomes fine-tunable on a single 48 GB GPU, with quality on instruction-following benchmarks within 1–2 % of full 16-bit fine-tuning. That's the gap from "lab project" to "your laptop's better GPU".
Quantization vs the adapter
The frozen base is dequantized on-the-fly during the forward pass back to BF16, multiplied with the activations, and the result is added to the BF16 LoRA path. Only the storage is 4-bit; compute is still BF16. That's why QLoRA loses very little quality vs full-precision LoRA — the math is the same, only the bookkeeping shrinks.
Exercise
A LoRA adapter writes the update as ΔW = B·A with B ∈ ℝ^{d×r} and A ∈ ℝ^{r×d}. For d = 4096 and r = 8, how many trainable parameters does LoRA store per weight matrix, compared to full fine-tuning?
Need a hint?
Full FT = d². LoRA = 2·d·r.
Exercise
A weight matrix is 1024 × 1024. You add a LoRA adapter of rank r = 8. Roughly what fraction (in percent) of the full d² parameters does LoRA store?
Need a hint?
LoRA params = 2·d·r. Compare to d² = 1024² ≈ 1.05M.
Exercise
What does the **Q** in QLoRA add on top of plain LoRA?
Need a hint?
It's about how the frozen base model is stored in memory.
Common mistakes
- Setting
rtoo high "to be safe." A rank-64 adapter on a 5k-example dataset will overfit fast. Start at 8 or 16 and only raise if validation loss plateaus. - Forgetting to scale by
α/r. Halvingrwithout halving the effective update magnitude silently changes your learning dynamics. - Attaching LoRA to only one projection.
q_projalone often under-performsq_proj+v_proj; budget two adapters before adding rank. - Merging adapters mid-training. Once you merge , you've baked the adapter into a new dense matrix — the parameter savings are gone until you re-extract.
- Using QLoRA for tiny models. The 4-bit overhead and dequantization cost only pay off for large bases (≳ 7B). For a 1.3B model, plain LoRA is faster.
Key takeaways
- LoRA learns a low-rank update on top of frozen base weights, cutting trainable parameters from to .
- Rank
rcontrols capacity; alphaαcontrols the effective magnitude. Effective update = . - Adapters are small files — hot-swappable, composable, ship-friendly.
- QLoRA quantizes the frozen base to 4-bit NF4 (and only the base — adapters stay in BF16), enabling 65B-model fine-tuning on a single consumer-grade GPU.
- Reach for full FT only when PEFT clearly underperforms; in 2025 LoRA/QLoRA is the default.
Related concepts
- Supervised Fine-Tuning: the loss you're still optimising; LoRA only changes which parameters move.
- Gradient descent: the optimiser still drives the (now-tiny) trainable subset.
- Regularization: low-rank constraints are themselves a form of inductive bias against overfitting.