Fine-Tuning & Alignment
Adapt foundation models to your domain and align them with human preferences — SFT, parameter-efficient fine-tuning (LoRA/QLoRA), reward models, RLHF, DPO, and quantization.
7 lessons
01Supervised Fine-Tuning & Instruction Tuning
When prompting isn't enough: update the model's weights on labeled examples. Loss, instruction datasets, chat templates, and the catastrophic-forgetting trap.
PEFT: LoRA & QLoRA
Parameter-efficient fine-tuning: freeze the base model, learn a low-rank update ΔW = B·A. Add 4-bit quantization (QLoRA) and a 65B model fits on one consumer GPU.
Reward Models
Turn pairwise human preferences into a scalar reward signal. The Bradley–Terry objective, a reward head on a frozen LM trunk, and the reward-hacking trap that breaks alignment pipelines.
Preference Tuning: RLHF & DPO
Two ways to turn a reward signal into an aligned policy: classical RLHF with PPO and a KL leash on the SFT model, or DPO — the same KL-regularised objective collapsed into one closed-form classification loss.
Model Merging & Quantization
Two cheap post-training tricks: combining the weights of multiple fine-tunes into one model, and shrinking the model itself by storing each weight in fewer bits.
Knowledge Distillation
Training a small student model to mimic a large teacher — soft labels, temperature scaling, and why 'dark knowledge' makes distilled models punch above their weight.
Fine-Tuning & Alignment Quiz
Test your understanding of supervised fine-tuning, LoRA/QLoRA, reward models, RLHF/DPO, quantization, and distillation.