Model Merging & Quantization
Fine-tuning was the expensive part. Once you have trained checkpoints, two near-free operations can make them dramatically more useful: merging several fine-tunes into a single model that inherits all their skills, and quantizing the weights so the model fits on smaller hardware.
Real-world examples: open-source chat models on Hugging Face's leaderboard frequently mix SLERP-merges of a base model with a math fine-tune and an instruction-following fine-tune; llama.cpp ships 4-bit GGUF builds of 70B models that run on a laptop; QLoRA fine-tunes a frozen 4-bit base and bolts on full-precision adapters.
Model merging
Once you have fine-tunes of the same base model, you can combine their weights into a single new model without any further training.
Weight averaging
The simplest merge is the arithmetic mean of weights:
This works only because the fine-tunes share an initialisation — they live in the same loss basin, so averaging stays near a minimum. Average two independently pretrained models and you get garbage; average two fine-tunes that diverged from the same base and you often get a usable model that inherits skills from both.
Task vectors
Reframe each fine-tune as a task vector — the displacement from the base:
Then merging becomes algebra: for some weights . Setting recovers plain averaging. Setting for one fine-tune subtracts a skill — a "task-vector negation" that, for example, removes a toxic-text fine-tune's effect.
SLERP
For two models, spherical linear interpolation moves along the great-circle arc rather than the chord:
When and have similar norm (typical for fine-tunes of the same base), SLERP keeps the merged model's weight norm closer to the originals than the chord average — empirically this preserves more of each parent's capability.
Why merging works at all
The loss surface of an over-parameterised model has wide, connected low-loss basins. Fine-tuning a shared base nudges the weights within the same basin, so the line between any two checkpoints often stays at low loss — a phenomenon called linear mode connectivity. Merging exploits that geometry, which is why the trick only survives when the parents share an initialisation.
Quantization
Pretrained weights are stored in FP32 (32 bits per parameter) or BF16 (16 bits). Inference doesn't need that much precision — most weights end up rounded to nearby values anyway during forward passes. Quantization stores each weight in fewer bits.
Uniform post-training quantization
Symmetric (absmax) quantization — the default for weights — maps the range to :
Affine (zero-point) quantization adds an integer zero-point so that the float zero maps exactly to an integer level, handling asymmetric distributions:
Symmetric is simpler (one scale per tensor/channel) and works well for weights whose distributions are roughly bell-shaped around zero. Affine is required for activations — ReLU outputs are non-negative, so their range is and a symmetric scheme would waste half the levels. W8A8 and W4A8 integer-matmul schemes use affine quantization on activations and symmetric on weights.
At inference the integers are de-quantized on the fly (or kept in integer form for INT8 matmul). The memory savings are exact:
| bits | levels | 7B-model memory |
|---|---|---|
| 32 (FP32) | 4.3 B | 26 GB |
| 16 (FP16/BF16) | 65 K | 13 GB |
| 8 (INT8) | 256 | 6.5 GB |
| 4 (INT4 / NF4) | 16 | 3.5 GB |
| 2 | 4 | 1.7 GB |
Post-training quantization: bit-width vs accuracy
Original FP32 weights (left) compressed to a discrete grid of 2^b levels (right). Lower bit-width shrinks the model — and adds quantization noise. The memory line assumes a 7B-parameter model.
Quantization-aware training (QAT)
Post-training quantization (PTQ) just rounds the trained weights. QAT simulates the rounding during training: the forward pass uses quantized weights, the backward pass uses the straight-through estimator (the round is treated as the identity for gradients), so the optimiser learns weights that survive rounding. QAT recovers more accuracy than PTQ at very low bit-widths (≤ 4) but costs a full training run.
The accuracy/size trade-off
Bit-width is the dial.
- INT8 is essentially free: < 1 % accuracy loss on most LLMs, half the memory of FP16.
- INT4 (especially with NF4 + double-quantization, as in QLoRA) loses a few points on the harder benchmarks. Still the sweet spot for consumer GPUs.
- INT2 / INT1 are research territory — usually only viable with QAT, knowledge distillation, or carefully chosen scales per layer.
QLoRA = quantization + LoRA
QLoRA is the headline application: freeze the base in 4-bit NF4, attach full-precision LoRA adapters, train only the adapters. Memory is dominated by the 4-bit base, so a 65B model fine-tunes on a single 48 GB GPU. The adapters stay full precision because they're tiny and bear all the gradient.
Exercise
A 7B-parameter model stored in FP16 takes ~13 GB. How many GB does the same model occupy when quantized to INT4?
Need a hint?
INT4 is 4 bits = 0.5 bytes per parameter.
Exercise
Which statement best describes the typical accuracy-vs-memory trade-off going from INT8 to INT4 on a modern LLM?
Need a hint?
INT8 is almost free; INT4 starts to bite but is still the production sweet spot.
Exercise
You have a base model θ_base, a math fine-tune θ_math, and a toxic-chat fine-tune θ_tox. You compute task vectors τ_math = θ_math − θ_base and τ_tox = θ_tox − θ_base. Which merged model gains math skill *and* explicitly removes the toxic-chat behaviour?
Need a hint?
Task-vector arithmetic: add the skill you want, subtract the one you don't.
Common mistakes
- Merging fine-tunes from different base models. They live in different loss basins; the average is meaningless.
- Using PTQ at INT2/INT1 without QAT. Accuracy collapses. Either step up to INT4 or pay for QAT.
- Treating quantization as a single global rounding. Modern schemes quantize per-channel or per-block (NF4 uses 64-element blocks with their own scales), which protects outliers far better than one global scale.
- Forgetting that activations also have precision. Weight-only quantization is the easy case; activation quantization (W8A8, W4A8) adds calibration steps and clipping decisions.
- Forgetting outliers. A handful of huge-magnitude weights dominate the dynamic range. Methods like SmoothQuant or AWQ rescale activations to push the dynamic range back onto weights where quantization is easier.
Key takeaways
- Merging combines fine-tunes that share a base by averaging weights, summing task vectors, or SLERPing — exploiting the linear mode connectivity of the shared loss basin.
- Task vectors turn merging into an algebra: add a skill by adding its vector, remove a skill by subtracting it.
- Uniform post-training quantization stores each weight in bits, mapping it to one of levels. Memory scales linearly with .
- INT8 is almost free; INT4 is the QLoRA sweet spot; INT2 and below need QAT or distillation to stay usable.
- QAT uses the straight-through estimator to learn rounding-robust weights at training time, recovering accuracy lost to aggressive PTQ.
Related concepts
- PEFT: LoRA & QLoRA: quantization powers the Q in QLoRA; this lesson explains the quantization scheme itself.
- Supervised Fine-Tuning: produces the checkpoints that get merged; quantization is applied after training.