Fine-Tuning & Alignment Quiz
Check your understanding of the alignment stack — when to fine-tune, how PEFT shrinks the training cost, how reward models capture preferences, how DPO replaces the PPO loop, and how quantization shrinks the deployed model.
Exercise
Which scenario is the *best* fit for supervised fine-tuning rather than prompt engineering or RAG?
Need a hint?
Fine-tuning changes behaviour, not knowledge.
Exercise
A weight matrix is 2048 × 2048. A LoRA adapter of rank r = 16 stores how many parameters, relative to full fine-tuning?
Need a hint?
Full = d². LoRA = 2·d·r.
Exercise
Why is the Bradley–Terry reward-model loss invariant to adding a constant to both $r_\theta(x, y_c)$ and $r_\theta(x, y_r)$?
Need a hint?
Look at what enters the sigmoid.
Exercise
Which combination correctly describes how DPO differs from classical RLHF with PPO?
Need a hint?
Think reward model, rollouts, and KL leash.
Exercise
A 13B-parameter model is currently served in FP16 (~26 GB). You quantize the weights to INT4. Roughly how much memory does the quantized model now use?
Need a hint?
INT4 = 0.5 bytes per parameter.
Exercise
In knowledge distillation, why does training the student on the teacher's full softened distribution beat training on hard labels alone?
Need a hint?
What does 'cat 78%, kitten 9%, dog 1.2%' tell you that 'cat' doesn't?
Exercise
A reward model turns pairwise human preferences into a scalar r(prompt, response) that later training maximizes. In the ML project loop, the reward model is which slot?
Need a hint?
It defines what 'good' means when there is no single correct label.
Exercise
Transfer test — a technique this course did NOT teach: an epsilon-greedy policy that occasionally takes a random action so an agent keeps discovering new states. Which slot of the ML project loop does exploration correspond to?
Need a hint?
It determines what experience — what data — the learner ever sees.