Knowledge DistillationOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Knowledge Distillation

Training a GPT-4-class model requires hundreds of billions of parameters and costs millions of dollars. Deploying it on every user request is even costlier. Knowledge distillation offers a way out: train a small, fast student model to mimic the output distribution of the large teacher, and the student often ends up far better than an identically-sized model trained from scratch.

Real-world examples: Phi-2/3 (Microsoft), Gemma-2 (Google), LLaMA distilled variants, DistilBERT (Hugging Face), and virtually every mobile- or edge-deployed LLM are distillation products. The core idea transfers beyond LLMs to any domain where you have a strong, expensive model and need a cheap one.

The core insight: soft labels carry more information

A hard label says "the answer is cat." A teacher's full output distribution over 50,000 tokens says "cat is 78 % likely, kitten 9 %, feline 4 %, dog 1.2 %..." — it encodes the relative similarity structure between classes. Hinton et al. called this dark knowledge: information the teacher has learned that isn't visible in the argmax.

Why dark knowledge matters

When training on hard labels, every wrong answer looks equally wrong. The teacher's distribution reveals which wrong answers are close — "kitten" and "cat" are nearly the same, while "airplane" and "cat" are not. The student learns this similarity structure, effectively getting richer supervision from each example.

The distillation loss

Distillation minimises the KL divergence between teacher and student probability distributions. For a single input xx with teacher output y^T\hat{y}_T and student output y^S\hat{y}_S over the vocabulary VV:

LKD=KL(y^T ∥ y^S)=∑i∈Vy^T(i)log⁡y^T(i)y^S(i)\mathcal{L}_{\text{KD}} = \text{KL}(\hat{y}_T \,\|\, \hat{y}_S) = \sum_{i \in V} \hat{y}_T^{(i)} \log \frac{\hat{y}_T^{(i)}}{\hat{y}_S^{(i)}}

In practice, a temperature τ>1\tau > 1 is applied to soften the distributions before computing the loss:

pi(τ)=exp⁡(zi/τ)∑jexp⁡(zj/τ)p_i^{(\tau)} = \frac{\exp(z_i / \tau)}{\sum_j \exp(z_j / \tau)}

where ziz_i are the raw logits. Higher τ\tau spreads probability mass across more tokens, making the soft labels more informative (and the KL target less spiky).

The full training objective typically blends distillation with standard cross-entropy on the true label:

L=α⋅τ2⋅LKD(T)  +  (1−α)⋅LCE\mathcal{L} = \alpha \cdot \tau^2 \cdot \mathcal{L}_{\text{KD}}(T) \;+\; (1-\alpha) \cdot \mathcal{L}_{\text{CE}}

The τ2\tau^2 factor compensates for the gradient scaling introduced by temperature (gradients are divided by τ\tau twice through the softmax). Common choices: α=0.5–0.9\alpha = 0.5\text{–}0.9, τ=2–10\tau = 2\text{–}10.

Temperature, and the τ² that keeps it honest

00.250.50.751cat0.852lynx0.085tiger0.052dog0.005fox0.004wolf·bear·deer·car·boat·× τ² — stays usableraw — collapsesτ = 101
p(top class)0.8521
mass outside the argmax0.1479
entropy (nats)0.5613
p(lynx) / p(tiger)1.648721
exp(0.5/τ)1.648721
gradient vs τ=1, raw1.0000×
gradient vs τ=1, ×τ²1.0000×

A teacher's ten logits, softened by τ. Ghost outlines are the τ = 1 distribution for comparison. Right: the norm of the distillation gradient with respect to the student's logits, relative to its value at τ = 1, with and without the τ² factor — log scale.

Three things are worth checking against the slider.

Softening reveals mass, never order. At τ=1\tau = 1 the top class holds 0.85210.8521 and everything else rounds to nothing; at τ=10\tau = 10 it holds 0.16850.1685 and the mass outside the argmax has gone from 0.1480.148 to 0.8320.832 (entropy 0.561→2.2550.561 \to 2.255 nats). The ranking is identical at every temperature (softmax is monotone in the logits), so temperature cannot change what the teacher believes, only how much of it survives into a loss.

What compresses is the ratios, exactly. For any two classes,

pipj=exp⁡ ⁣(zi−zjτ)\frac{p_i}{p_j} = \exp\!\left(\frac{z_i - z_j}{\tau}\right)

The panel prints p(lynx)/p(tiger) next to exp⁡(0.5/τ)\exp(0.5/\tau) and they agree to six decimals: 1.6487211.648721 at τ=1\tau=1, 1.1331481.133148 at τ=4\tau=4, 1.0512711.051271 at τ=10\tau=10. "Dark knowledge" is not a metaphor. It is those ratios being lifted out of floating-point irrelevance so a cross-entropy can see them.

And the τ2\tau^2 is not a fudge factor. The KD gradient with respect to the student's logits is (pstudent−pteacher)/τ(p_{\text{student}} - p_{\text{teacher}})/\tau; both the difference and the 1/τ1/\tau shrink as you soften, so the raw gradient norm falls 118× across τ=1→10\tau = 1 \to 10 (0.383 to 0.0032). Without the correction, raising the temperature would quietly turn the distillation term off. With it, the norm holds at 0.383 → 0.324 and α\alpha keeps meaning what you set it to.

Why not just use a lower temperature for the teacher?

You could. The standard choice τ=1\tau = 1 recovers hard labels in the limit. Using τ>1\tau > 1 spreads the teacher's distribution, making near-misses more visible to the student. Too high (τ≫10\tau \gg 10) and the distribution becomes nearly uniform — no signal. The sweet spot depends on the model and task and is usually found by validation.

Python
import torch
import torch.nn.functional as F

def distillation_loss(student_logits, teacher_logits, labels, tau=4.0, alpha=0.7):
    # Soft targets — both teacher and student softened at temperature τ
    soft_student = F.log_softmax(student_logits / tau, dim=-1)
    soft_teacher = F.softmax(teacher_logits / tau, dim=-1)

    # KL divergence (τ² compensates gradient scaling)
    kd_loss = F.kl_div(soft_student, soft_teacher, reduction="batchmean") * (tau ** 2)

    # Hard-label cross-entropy
    ce_loss = F.cross_entropy(student_logits, labels)

    return alpha * kd_loss + (1 - alpha) * ce_loss

Types of distillation

Output (response) distillation: the student learns from the teacher's final token probability distribution, as above. Cheapest to run; the teacher is frozen and queried offline.

Feature (intermediate) distillation: the student is also trained to match the teacher's hidden states or attention maps at intermediate layers. Adds more signal but requires architectural alignment (same hidden dimensions or an adapter layer).

Sequence-level distillation: for generative models, the teacher generates complete responses for each prompt; the student trains on those responses via standard cross-entropy. This is the most common form in LLM distillation: generate a synthetic dataset of (prompt, teacher_response) pairs, then SFT the student on them.

TypeTeacher outputs usedCostSignal richness
ResponseFinal logits / probabilitiesLowMedium
FeatureIntermediate activationsHighHigh
Sequence-levelFull generated responsesMediumLow (hard labels)

Worked example: DistilBERT

DistilBERT reduces BERT-base (110 M params, 12 layers) to a 6-layer, 66 M-parameter student with three simultaneous objectives:

  1. Soft MLM loss: KL divergence between student and teacher on masked token predictions.
  2. Hard MLM loss: standard cross-entropy against the true masked token.
  3. Cosine embedding loss: student's hidden states pushed to align with teacher's at each layer.

Result: 97 % of BERT's performance on GLUE at 60 % the size and 60 % the inference time. The multi-objective approach is now the standard template for BERT-family distillation.

LLM-specific patterns

For instruction-following LLMs, sequence-level distillation dominates because:

  1. Logit access is often unavailable — the teacher may be a closed API (GPT-4, Claude) returning only text, not probabilities.
  2. Vocabulary mismatch: teacher and student often use different tokenizers, making token-level KD non-trivial.
  3. Long responses: a full response provides enough supervision even as a hard-label sequence.

The Phi-3 family exemplifies this: the training set is "textbook-quality" synthetic data generated by GPT-4, and the student is trained on it with standard next-token prediction — conceptually simple sequence-level distillation.

Distillation inherits teacher errors

The student learns what the teacher does, including its biases, refusals, and hallucinations. If the teacher is miscalibrated or politically biased, the student may amplify those flaws. Always evaluate the distilled student independently, not just against the teacher's outputs.

Distillation vs. other compression techniques

MethodChangesRecovers accuracy?Notes
DistillationTrains a smaller architectureYes (with enough data)Best quality at fixed size
QuantizationReduces bit-widthMostly (INT8), partially (INT4)No retraining for PTQ
PruningRemoves weights/headsPartiallyNeeds fine-tuning after
LoRAAdds low-rank adaptersN/A (fine-tuning, not compression)—

These techniques compose: QLoRA distillation (distill to a small model, then quantize to INT4) is common for mobile deployment.

Exercise

Why do 'soft labels' from a teacher model provide richer supervision than one-hot hard labels?

Need a hint?

Think about what information is encoded in the non-argmax probabilities.

Exercise

In knowledge distillation, a temperature τ > 1 is applied to both the teacher and student logits before computing the KL divergence loss. What does raising τ achieve?

Need a hint?

Consider what happens to the softmax distribution as temperature increases.

Exercise

You need to deploy a 1 B-parameter LLM. You can either (a) train a 1 B model from scratch on the same data or (b) distill a 70 B teacher into a 1 B student. With the same compute budget, which approach typically produces a better model?

Need a hint?

Think about what information each training signal provides per example.

Common mistakes

  1. Using τ=1\tau = 1: at temperature 1, the teacher's argmax dominates; you lose most of the dark knowledge that makes distillation valuable.
  2. Skipping the hard-label term: pure KD can overfit to the teacher's errors; the CE term anchors the student to ground truth.
  3. Forgetting the τ2\tau^2 gradient correction: omitting it means the KD gradient is scaled down by τ2\tau^2 relative to the CE gradient, so α\alpha effectively means something different from what you intend.
  4. Assuming closed-API teachers can't be distilled: sequence-level distillation only needs generated text, not logits.

Key takeaways

  • Soft labels from the teacher encode similarity structure between outputs ("dark knowledge") that hard labels cannot.
  • The distillation loss is KL divergence between teacher and student distributions, applied at temperature τ>1\tau > 1 to amplify near-misses.
  • The full objective blends LKD\mathcal{L}_{\text{KD}} (soft) and LCE\mathcal{L}_{\text{CE}} (hard) with weight α\alpha.
  • Sequence-level distillation (student trains on teacher-generated text) is dominant for LLMs because logit access and vocabulary alignment are often unavailable.
  • Distillation composes with quantization and pruning — it's the quality floor, the others compress further.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelated
Knowledge DistillationAttention & TransformersReinforcement LearningPEFT: LoRA & QLoRAModel Merging & Quantizat…Supervised Fine-Tuning & …