Supervised Fine-Tuning & Instruction TuningOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Supervised Fine-Tuning & Instruction Tuning

Prompting and retrieval keep the weights frozen. Supervised fine-tuning (SFT) actually updates them. You hand the model a dataset of input/output pairs and continue training it with the same next-token cross-entropy loss it was pretrained with, but now on your distribution.

This course is one trip around the ML loop

Fine-tuning and alignment are the cleanest whole-course showcase of the ML project loop — each lesson is a different slot, applied to a frozen foundation model:

LessonSlot it changes
SFT (this lesson)data: curate input/output pairs on your distribution
PEFT / LoRAhypothesis space: a small low-rank adapter is the trainable model
Reward modelsobjective: turn human preference into a scalar to optimize
RLHF / DPOoptimization: search the policy for high-reward responses
Merging / quantization / distillationdeployment feedback: make the trained model cheap and serveable

Read the course with that map in hand and each technique stops being a separate trick — it's a change to one slot of a loop you already know.

Deep dive
The ML Project Loop
Includes runnable notebook

Real-world examples: a customer-support assistant that always replies in your brand voice; a coding model specialised for your internal framework; a medical-summarisation model trained on de-identified discharge notes; an instruction-tuned chat model built from a base LLM and a curated instruction dataset.

When to fine-tune vs prompt vs RAG

Fine-tuning is the most expensive lever — try the cheaper ones first.

LeverChanges weights?Best forCost
Prompt engineeringNoSteering, formatting, simple tasks$
RAGNoFresh / large knowledge bases$$
Fine-tuningYesNew skills, style, domain language, structured output$$$

Rule of thumb

If a careful prompt with 5–10 well-chosen examples gets you to 90 % of where you need to go, fine-tuning is rarely worth it. Reach for SFT when the behaviour itself (tone, structure, reasoning steps) needs to change, not the knowledge.

The SFT loss

For every training pair (x,y)(x, y) (prompt xx, target completion y=(y1,…,yT)y = (y_1, \dots, y_T)), the loss is plain token-level cross-entropy on the target tokens:

LSFT=−∑t=1Tlog⁡pθ(yt∣x, y<t)\mathcal{L}_{\text{SFT}} = -\sum_{t=1}^{T} \log p_\theta(y_t \mid x,\, y_{<t})

The prompt tokens are not scored; their loss is masked out. The model is only penalised for the response it generates, conditioned on the prompt.

Gradient descent on f(x) = x²

weight xloss
step0
x-4.200
loss17.640

Each step moves the ball by −learning-rate × gradient. Push the learning rate past 1.0 to watch it diverge.

Instruction datasets

Pretraining sees raw web text. Instruction tuning shows the model what helpful looks like by training on examples of (instruction → desired response). High-quality datasets share a few traits:

  • Coverage: broad task types (summarisation, extraction, rewriting, Q&A, code).
  • Diversity: varied phrasings, lengths, and difficulty so the model doesn't overfit to one template.
  • Cleanliness: fewer than 10 000 carefully curated examples often outperform millions of noisy ones (the "less is more" finding from LIMA, Alpaca-cleaned, and similar work).

Chat templates

A modern chat model expects messages in a structured format with special tokens marking role boundaries — typically something like <|user|>, <|assistant|>, <|system|>. The fine-tuning pipeline must apply the same template the inference server will apply at runtime, or the model will see a distribution it was never trained on.

Text
<|system|>You are a helpful, terse assistant.<|end|>
<|user|>Summarise the email in one sentence: ...<|end|>
<|assistant|>{response — this is what the loss is computed on}<|end|>

Only the highlighted assistant tokens contribute to the loss; the system + user tokens are masked.

Catastrophic forgetting

Update every weight aggressively on a narrow dataset and the model forgets abilities it had before — fluent reasoning, multilingual coverage, code generation. The classic symptoms:

  1. The fine-tuned model nails your task but loses its general chat ability.
  2. Held-out benchmarks (MMLU, HumanEval) drop noticeably from the base model.
  3. The model parrots your training format on inputs that didn't ask for it.

Mitigations:

  • Small learning rate (10−510^{-5} to 10−610^{-6}): much smaller than pretraining.
  • Few epochs (often just 1–3 passes).
  • Parameter-efficient methods: train tiny adapters and freeze the base (next lesson: PEFT, LoRA & QLoRA).
  • Mix in general-purpose data so the model keeps its old skills.

Exercise

Your assistant occasionally cites stale prices because the product catalog changes weekly. Which lever should you reach for first?

Need a hint?

Is the gap knowledge or behaviour?

Exercise

During SFT on (prompt, response) pairs, which tokens contribute to the cross-entropy loss?

Need a hint?

Think about what the model is being trained to *produce*.

Exercise

After fine-tuning a base LLM on 50k legal-summarisation examples for 10 epochs, its general MMLU score drops 12 points. What is the most likely cause?

Need a hint?

Aggressive updates on a narrow distribution.

Common mistakes

  1. Fine-tuning to inject facts. Use RAG — fine-tuning is for behaviour, not knowledge.
  2. Forgetting to mask the prompt. If you train the model to predict its own input, you'll waste capacity and degrade generation.
  3. Mismatched chat template between training and serving — silently destroys quality.
  4. Too many epochs. 5+ epochs on a small dataset is a one-way ticket to overfitting and forgetting.
  5. No held-out general-capability eval. You'll only learn about catastrophic forgetting after deploy.

Key takeaways

  • SFT continues pretraining on (input, target) pairs using the same next-token cross-entropy, but only on the target tokens.
  • Pick SFT when behaviour must change; prefer prompting + RAG when knowledge is what you need.
  • Instruction tuning is just SFT on (instruction → desired response) pairs, with a consistent chat template.
  • Aggressive fine-tuning causes catastrophic forgetting: use small learning rates, few epochs, and parameter-efficient methods.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelatedGo deeper
Supervised Fine-Tuning & Instruct…Attention & TransformersReinforcement LearningFoundation Models & Scali…Evaluating AI SystemsRetrieval-Augmented Gener…Gradient DescentPrompt Engineering & In-C…PEFT: LoRA & QLoRAThe ML Project Loop