Knowledge Distillation
Training a GPT-4-class model requires hundreds of billions of parameters and costs millions of dollars. Deploying it on every user request is even costlier. Knowledge distillation offers a way out: train a small, fast student model to mimic the output distribution of the large teacher, and the student often ends up far better than an identically-sized model trained from scratch.
Real-world examples: Phi-2/3 (Microsoft), Gemma-2 (Google), LLaMA distilled variants, DistilBERT (Hugging Face), and virtually every mobile- or edge-deployed LLM are distillation products. The core idea transfers beyond LLMs to any domain where you have a strong, expensive model and need a cheap one.
The core insight: soft labels carry more information
A hard label says "the answer is cat." A teacher's full output distribution over 50,000 tokens says "cat is 78 % likely, kitten 9 %, feline 4 %, dog 1.2 %..." — it encodes the relative similarity structure between classes. Hinton et al. called this dark knowledge: information the teacher has learned that isn't visible in the argmax.
Why dark knowledge matters
When training on hard labels, every wrong answer looks equally wrong. The teacher's distribution reveals which wrong answers are close — "kitten" and "cat" are nearly the same, while "airplane" and "cat" are not. The student learns this similarity structure, effectively getting richer supervision from each example.
The distillation loss
Distillation minimises the KL divergence between teacher and student probability distributions. For a single input with teacher output and student output over the vocabulary :
In practice, a temperature is applied to soften the distributions before computing the loss:
where are the raw logits. Higher spreads probability mass across more tokens, making the soft labels more informative (and the KL target less spiky).
The full training objective typically blends distillation with standard cross-entropy on the true label:
The factor compensates for the gradient scaling introduced by temperature (gradients are divided by twice through the softmax). Common choices: , .
Temperature, and the τ² that keeps it honest
A teacher's ten logits, softened by τ. Ghost outlines are the τ = 1 distribution for comparison. Right: the norm of the distillation gradient with respect to the student's logits, relative to its value at τ = 1, with and without the τ² factor — log scale.
Three things are worth checking against the slider.
Softening reveals mass, never order. At the top class holds and everything else rounds to nothing; at it holds and the mass outside the argmax has gone from to (entropy nats). The ranking is identical at every temperature (softmax is monotone in the logits), so temperature cannot change what the teacher believes, only how much of it survives into a loss.
What compresses is the ratios, exactly. For any two classes,
The panel prints p(lynx)/p(tiger) next to and they agree to six decimals: at , at , at . "Dark knowledge" is not a metaphor. It is those ratios being lifted out of floating-point irrelevance so a cross-entropy can see them.
And the is not a fudge factor. The KD gradient with respect to the student's logits is ; both the difference and the shrink as you soften, so the raw gradient norm falls 118× across (0.383 to 0.0032). Without the correction, raising the temperature would quietly turn the distillation term off. With it, the norm holds at 0.383 → 0.324 and keeps meaning what you set it to.
Why not just use a lower temperature for the teacher?
You could. The standard choice recovers hard labels in the limit. Using spreads the teacher's distribution, making near-misses more visible to the student. Too high () and the distribution becomes nearly uniform — no signal. The sweet spot depends on the model and task and is usually found by validation.
import torch
import torch.nn.functional as F
def distillation_loss(student_logits, teacher_logits, labels, tau=4.0, alpha=0.7):
# Soft targets — both teacher and student softened at temperature τ
soft_student = F.log_softmax(student_logits / tau, dim=-1)
soft_teacher = F.softmax(teacher_logits / tau, dim=-1)
# KL divergence (τ² compensates gradient scaling)
kd_loss = F.kl_div(soft_student, soft_teacher, reduction="batchmean") * (tau ** 2)
# Hard-label cross-entropy
ce_loss = F.cross_entropy(student_logits, labels)
return alpha * kd_loss + (1 - alpha) * ce_loss
Types of distillation
Output (response) distillation: the student learns from the teacher's final token probability distribution, as above. Cheapest to run; the teacher is frozen and queried offline.
Feature (intermediate) distillation: the student is also trained to match the teacher's hidden states or attention maps at intermediate layers. Adds more signal but requires architectural alignment (same hidden dimensions or an adapter layer).
Sequence-level distillation: for generative models, the teacher generates complete responses for each prompt; the student trains on those responses via standard cross-entropy. This is the most common form in LLM distillation: generate a synthetic dataset of (prompt, teacher_response) pairs, then SFT the student on them.
| Type | Teacher outputs used | Cost | Signal richness |
|---|---|---|---|
| Response | Final logits / probabilities | Low | Medium |
| Feature | Intermediate activations | High | High |
| Sequence-level | Full generated responses | Medium | Low (hard labels) |
Worked example: DistilBERT
DistilBERT reduces BERT-base (110 M params, 12 layers) to a 6-layer, 66 M-parameter student with three simultaneous objectives:
- Soft MLM loss: KL divergence between student and teacher on masked token predictions.
- Hard MLM loss: standard cross-entropy against the true masked token.
- Cosine embedding loss: student's hidden states pushed to align with teacher's at each layer.
Result: 97 % of BERT's performance on GLUE at 60 % the size and 60 % the inference time. The multi-objective approach is now the standard template for BERT-family distillation.
LLM-specific patterns
For instruction-following LLMs, sequence-level distillation dominates because:
- Logit access is often unavailable — the teacher may be a closed API (GPT-4, Claude) returning only text, not probabilities.
- Vocabulary mismatch: teacher and student often use different tokenizers, making token-level KD non-trivial.
- Long responses: a full response provides enough supervision even as a hard-label sequence.
The Phi-3 family exemplifies this: the training set is "textbook-quality" synthetic data generated by GPT-4, and the student is trained on it with standard next-token prediction — conceptually simple sequence-level distillation.
Distillation inherits teacher errors
The student learns what the teacher does, including its biases, refusals, and hallucinations. If the teacher is miscalibrated or politically biased, the student may amplify those flaws. Always evaluate the distilled student independently, not just against the teacher's outputs.
Distillation vs. other compression techniques
| Method | Changes | Recovers accuracy? | Notes |
|---|---|---|---|
| Distillation | Trains a smaller architecture | Yes (with enough data) | Best quality at fixed size |
| Quantization | Reduces bit-width | Mostly (INT8), partially (INT4) | No retraining for PTQ |
| Pruning | Removes weights/heads | Partially | Needs fine-tuning after |
| LoRA | Adds low-rank adapters | N/A (fine-tuning, not compression) | — |
These techniques compose: QLoRA distillation (distill to a small model, then quantize to INT4) is common for mobile deployment.
Exercise
Why do 'soft labels' from a teacher model provide richer supervision than one-hot hard labels?
Need a hint?
Think about what information is encoded in the non-argmax probabilities.
Exercise
In knowledge distillation, a temperature τ > 1 is applied to both the teacher and student logits before computing the KL divergence loss. What does raising τ achieve?
Need a hint?
Consider what happens to the softmax distribution as temperature increases.
Exercise
You need to deploy a 1 B-parameter LLM. You can either (a) train a 1 B model from scratch on the same data or (b) distill a 70 B teacher into a 1 B student. With the same compute budget, which approach typically produces a better model?
Need a hint?
Think about what information each training signal provides per example.
Common mistakes
- Using : at temperature 1, the teacher's argmax dominates; you lose most of the dark knowledge that makes distillation valuable.
- Skipping the hard-label term: pure KD can overfit to the teacher's errors; the CE term anchors the student to ground truth.
- Forgetting the gradient correction: omitting it means the KD gradient is scaled down by relative to the CE gradient, so effectively means something different from what you intend.
- Assuming closed-API teachers can't be distilled: sequence-level distillation only needs generated text, not logits.
Key takeaways
- Soft labels from the teacher encode similarity structure between outputs ("dark knowledge") that hard labels cannot.
- The distillation loss is KL divergence between teacher and student distributions, applied at temperature to amplify near-misses.
- The full objective blends (soft) and (hard) with weight .
- Sequence-level distillation (student trains on teacher-generated text) is dominant for LLMs because logit access and vocabulary alignment are often unavailable.
- Distillation composes with quantization and pruning — it's the quality floor, the others compress further.
Related concepts
- PEFT & LoRA: parameter-efficient fine-tuning that keeps the student small during adaptation
- Model Merging & Quantization: compression that stacks on top of distillation
- Supervised Fine-Tuning: the same next-token training loop used by sequence-level distillation