Supervised Fine-Tuning & Instruction Tuning
Prompting and retrieval keep the weights frozen. Supervised fine-tuning (SFT) actually updates them. You hand the model a dataset of input/output pairs and continue training it with the same next-token cross-entropy loss it was pretrained with, but now on your distribution.
This course is one trip around the ML loop
Fine-tuning and alignment are the cleanest whole-course showcase of the ML project loop — each lesson is a different slot, applied to a frozen foundation model:
| Lesson | Slot it changes |
|---|---|
| SFT (this lesson) | data: curate input/output pairs on your distribution |
| PEFT / LoRA | hypothesis space: a small low-rank adapter is the trainable model |
| Reward models | objective: turn human preference into a scalar to optimize |
| RLHF / DPO | optimization: search the policy for high-reward responses |
| Merging / quantization / distillation | deployment feedback: make the trained model cheap and serveable |
Read the course with that map in hand and each technique stops being a separate trick — it's a change to one slot of a loop you already know.
Real-world examples: a customer-support assistant that always replies in your brand voice; a coding model specialised for your internal framework; a medical-summarisation model trained on de-identified discharge notes; an instruction-tuned chat model built from a base LLM and a curated instruction dataset.
When to fine-tune vs prompt vs RAG
Fine-tuning is the most expensive lever — try the cheaper ones first.
| Lever | Changes weights? | Best for | Cost |
|---|---|---|---|
| Prompt engineering | No | Steering, formatting, simple tasks | $ |
| RAG | No | Fresh / large knowledge bases | $$ |
| Fine-tuning | Yes | New skills, style, domain language, structured output | $$$ |
Rule of thumb
If a careful prompt with 5–10 well-chosen examples gets you to 90 % of where you need to go, fine-tuning is rarely worth it. Reach for SFT when the behaviour itself (tone, structure, reasoning steps) needs to change, not the knowledge.
The SFT loss
For every training pair (prompt , target completion ), the loss is plain token-level cross-entropy on the target tokens:
The prompt tokens are not scored; their loss is masked out. The model is only penalised for the response it generates, conditioned on the prompt.
Gradient descent on f(x) = x²
Each step moves the ball by −learning-rate × gradient. Push the learning rate past 1.0 to watch it diverge.
Instruction datasets
Pretraining sees raw web text. Instruction tuning shows the model what helpful looks like by training on examples of (instruction → desired response). High-quality datasets share a few traits:
- Coverage: broad task types (summarisation, extraction, rewriting, Q&A, code).
- Diversity: varied phrasings, lengths, and difficulty so the model doesn't overfit to one template.
- Cleanliness: fewer than 10 000 carefully curated examples often outperform millions of noisy ones (the "less is more" finding from LIMA, Alpaca-cleaned, and similar work).
Chat templates
A modern chat model expects messages in a structured format with special tokens marking role boundaries — typically something like <|user|>, <|assistant|>, <|system|>. The fine-tuning pipeline must apply the same template the inference server will apply at runtime, or the model will see a distribution it was never trained on.
<|system|>You are a helpful, terse assistant.<|end|>
<|user|>Summarise the email in one sentence: ...<|end|>
<|assistant|>{response — this is what the loss is computed on}<|end|>
Only the highlighted assistant tokens contribute to the loss; the system + user tokens are masked.
Catastrophic forgetting
Update every weight aggressively on a narrow dataset and the model forgets abilities it had before — fluent reasoning, multilingual coverage, code generation. The classic symptoms:
- The fine-tuned model nails your task but loses its general chat ability.
- Held-out benchmarks (MMLU, HumanEval) drop noticeably from the base model.
- The model parrots your training format on inputs that didn't ask for it.
Mitigations:
- Small learning rate ( to ): much smaller than pretraining.
- Few epochs (often just 1–3 passes).
- Parameter-efficient methods: train tiny adapters and freeze the base (next lesson: PEFT, LoRA & QLoRA).
- Mix in general-purpose data so the model keeps its old skills.
Exercise
Your assistant occasionally cites stale prices because the product catalog changes weekly. Which lever should you reach for first?
Need a hint?
Is the gap knowledge or behaviour?
Exercise
During SFT on (prompt, response) pairs, which tokens contribute to the cross-entropy loss?
Need a hint?
Think about what the model is being trained to *produce*.
Exercise
After fine-tuning a base LLM on 50k legal-summarisation examples for 10 epochs, its general MMLU score drops 12 points. What is the most likely cause?
Need a hint?
Aggressive updates on a narrow distribution.
Common mistakes
- Fine-tuning to inject facts. Use RAG — fine-tuning is for behaviour, not knowledge.
- Forgetting to mask the prompt. If you train the model to predict its own input, you'll waste capacity and degrade generation.
- Mismatched chat template between training and serving — silently destroys quality.
- Too many epochs. 5+ epochs on a small dataset is a one-way ticket to overfitting and forgetting.
- No held-out general-capability eval. You'll only learn about catastrophic forgetting after deploy.
Key takeaways
- SFT continues pretraining on (input, target) pairs using the same next-token cross-entropy, but only on the target tokens.
- Pick SFT when behaviour must change; prefer prompting + RAG when knowledge is what you need.
- Instruction tuning is just SFT on (instruction → desired response) pairs, with a consistent chat template.
- Aggressive fine-tuning causes catastrophic forgetting: use small learning rates, few epochs, and parameter-efficient methods.
Related concepts
- Gradient descent: the optimiser SFT relies on
- Building with LLMs: the cheaper levers (prompting, few-shot) you should try first
- Retrieval-Augmented Generation: the right tool when the gap is knowledge, not behaviour