Model Merging & QuantizationOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Model Merging & Quantization

Fine-tuning was the expensive part. Once you have trained checkpoints, two near-free operations can make them dramatically more useful: merging several fine-tunes into a single model that inherits all their skills, and quantizing the weights so the model fits on smaller hardware.

Real-world examples: open-source chat models on Hugging Face's leaderboard frequently mix SLERP-merges of a base model with a math fine-tune and an instruction-following fine-tune; llama.cpp ships 4-bit GGUF builds of 70B models that run on a laptop; QLoRA fine-tunes a frozen 4-bit base and bolts on full-precision adapters.

Model merging

Once you have KK fine-tunes of the same base model, you can combine their weights into a single new model without any further training.

Weight averaging

The simplest merge is the arithmetic mean of weights:

θmerge=1K∑k=1Kθk.\theta_{\text{merge}} = \frac{1}{K} \sum_{k=1}^{K} \theta_k.

This works only because the fine-tunes share an initialisation — they live in the same loss basin, so averaging stays near a minimum. Average two independently pretrained models and you get garbage; average two fine-tunes that diverged from the same base and you often get a usable model that inherits skills from both.

Task vectors

Reframe each fine-tune as a task vector — the displacement from the base:

τk=θk−θbase.\tau_k = \theta_k - \theta_{\text{base}}.

Then merging becomes algebra: θmerge=θbase+∑kαkτk\theta_{\text{merge}} = \theta_{\text{base}} + \sum_k \alpha_k \tau_k for some weights αk\alpha_k. Setting αk=1/K\alpha_k = 1/K recovers plain averaging. Setting αk=−1\alpha_k = -1 for one fine-tune subtracts a skill — a "task-vector negation" that, for example, removes a toxic-text fine-tune's effect.

SLERP

For two models, spherical linear interpolation moves along the great-circle arc rather than the chord:

θSLERP(t)=sin⁡((1−t)Ω)sin⁡Ωθ1+sin⁡(tΩ)sin⁡Ωθ2,cos⁡Ω=θ1⋅θ2∥θ1∥ ∥θ2∥.\theta_{\text{SLERP}}(t) = \frac{\sin((1-t)\Omega)}{\sin\Omega}\theta_1 + \frac{\sin(t\Omega)}{\sin\Omega}\theta_2, \qquad \cos\Omega = \frac{\theta_1 \cdot \theta_2}{\|\theta_1\|\,\|\theta_2\|}.

When θ1\theta_1 and θ2\theta_2 have similar norm (typical for fine-tunes of the same base), SLERP keeps the merged model's weight norm closer to the originals than the chord average — empirically this preserves more of each parent's capability.

Why merging works at all

The loss surface of an over-parameterised model has wide, connected low-loss basins. Fine-tuning a shared base nudges the weights within the same basin, so the line between any two checkpoints often stays at low loss — a phenomenon called linear mode connectivity. Merging exploits that geometry, which is why the trick only survives when the parents share an initialisation.

Quantization

Pretrained weights are stored in FP32 (32 bits per parameter) or BF16 (16 bits). Inference doesn't need that much precision — most weights end up rounded to nearby values anyway during forward passes. Quantization stores each weight in fewer bits.

Uniform post-training quantization

Symmetric (absmax) quantization — the default for weights — maps the range [−wmax⁡,wmax⁡][-w_{\max}, w_{\max}] to [−(2b−1−1),2b−1−1][-(2^{b-1}-1), 2^{b-1}-1]:

q(w)=round ⁣(wΔ)Δ,Δ=2wmax⁡2b−1.q(w) = \text{round}\!\left(\frac{w}{\Delta}\right)\Delta, \qquad \Delta = \frac{2 w_{\max}}{2^b - 1}.

Affine (zero-point) quantization adds an integer zero-point zz so that the float zero maps exactly to an integer level, handling asymmetric distributions:

q(w)=round ⁣(wΔ+z),wapprox=Δ (q−z).q(w) = \text{round}\!\left(\frac{w}{\Delta} + z\right), \qquad w_{\text{approx}} = \Delta\,(q - z).

Symmetric is simpler (one scale per tensor/channel) and works well for weights whose distributions are roughly bell-shaped around zero. Affine is required for activations — ReLU outputs are non-negative, so their range is [0,xmax⁡][0, x_{\max}] and a symmetric scheme would waste half the levels. W8A8 and W4A8 integer-matmul schemes use affine quantization on activations and symmetric on weights.

At inference the integers are de-quantized on the fly (or kept in integer form for INT8 matmul). The memory savings are exact:

bitslevels7B-model memory
32 (FP32)4.3 B26 GB
16 (FP16/BF16)65 K13 GB
8 (INT8)2566.5 GB
4 (INT4 / NF4)163.5 GB
241.7 GB

Post-training quantization: bit-width vs accuracy

FP32 weights→8-bit weights
bit-width8-bit
levels256
7B model memory6.5 GB
quantization error0.9%

Original FP32 weights (left) compressed to a discrete grid of 2^b levels (right). Lower bit-width shrinks the model — and adds quantization noise. The memory line assumes a 7B-parameter model.

Quantization-aware training (QAT)

Post-training quantization (PTQ) just rounds the trained weights. QAT simulates the rounding during training: the forward pass uses quantized weights, the backward pass uses the straight-through estimator (the round is treated as the identity for gradients), so the optimiser learns weights that survive rounding. QAT recovers more accuracy than PTQ at very low bit-widths (≤ 4) but costs a full training run.

The accuracy/size trade-off

Bit-width is the dial.

  • INT8 is essentially free: < 1 % accuracy loss on most LLMs, half the memory of FP16.
  • INT4 (especially with NF4 + double-quantization, as in QLoRA) loses a few points on the harder benchmarks. Still the sweet spot for consumer GPUs.
  • INT2 / INT1 are research territory — usually only viable with QAT, knowledge distillation, or carefully chosen scales per layer.

QLoRA = quantization + LoRA

QLoRA is the headline application: freeze the base in 4-bit NF4, attach full-precision LoRA adapters, train only the adapters. Memory is dominated by the 4-bit base, so a 65B model fine-tunes on a single 48 GB GPU. The adapters stay full precision because they're tiny and bear all the gradient.

Exercise

A 7B-parameter model stored in FP16 takes ~13 GB. How many GB does the same model occupy when quantized to INT4?

0GB5.00GB10GB
Need a hint?

INT4 is 4 bits = 0.5 bytes per parameter.

Exercise

Which statement best describes the typical accuracy-vs-memory trade-off going from INT8 to INT4 on a modern LLM?

Need a hint?

INT8 is almost free; INT4 starts to bite but is still the production sweet spot.

Exercise

You have a base model θ_base, a math fine-tune θ_math, and a toxic-chat fine-tune θ_tox. You compute task vectors τ_math = θ_math − θ_base and τ_tox = θ_tox − θ_base. Which merged model gains math skill *and* explicitly removes the toxic-chat behaviour?

Need a hint?

Task-vector arithmetic: add the skill you want, subtract the one you don't.

Common mistakes

  1. Merging fine-tunes from different base models. They live in different loss basins; the average is meaningless.
  2. Using PTQ at INT2/INT1 without QAT. Accuracy collapses. Either step up to INT4 or pay for QAT.
  3. Treating quantization as a single global rounding. Modern schemes quantize per-channel or per-block (NF4 uses 64-element blocks with their own scales), which protects outliers far better than one global scale.
  4. Forgetting that activations also have precision. Weight-only quantization is the easy case; activation quantization (W8A8, W4A8) adds calibration steps and clipping decisions.
  5. Forgetting outliers. A handful of huge-magnitude weights dominate the dynamic range. Methods like SmoothQuant or AWQ rescale activations to push the dynamic range back onto weights where quantization is easier.

Key takeaways

  • Merging combines fine-tunes that share a base by averaging weights, summing task vectors, or SLERPing — exploiting the linear mode connectivity of the shared loss basin.
  • Task vectors turn merging into an algebra: add a skill by adding its vector, remove a skill by subtracting it.
  • Uniform post-training quantization stores each weight in bb bits, mapping it to one of 2b2^b levels. Memory scales linearly with bb.
  • INT8 is almost free; INT4 is the QLoRA sweet spot; INT2 and below need QAT or distillation to stay usable.
  • QAT uses the straight-through estimator to learn rounding-robust weights at training time, recovering accuracy lost to aggressive PTQ.

Related concepts

  • PEFT: LoRA & QLoRA: quantization powers the Q in QLoRA; this lesson explains the quantization scheme itself.
  • Supervised Fine-Tuning: produces the checkpoints that get merged; quantization is applied after training.

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelated
Model Merging & QuantizationAttention & TransformersReinforcement LearningDeployment Patterns & Mod…Optimizing LLM InferencePEFT: LoRA & QLoRASupervised Fine-Tuning & …Knowledge Distillation