Fine-Tuning & Alignment Quiz

Fine-Tuning & Alignment Quiz

Check your understanding of the alignment stack — when to fine-tune, how PEFT shrinks the training cost, how reward models capture preferences, how DPO replaces the PPO loop, and how quantization shrinks the deployed model.

Exercise

Which scenario is the *best* fit for supervised fine-tuning rather than prompt engineering or RAG?

Need a hint?

Fine-tuning changes behaviour, not knowledge.

Exercise

A weight matrix is 2048 × 2048. A LoRA adapter of rank r = 16 stores how many parameters, relative to full fine-tuning?

Need a hint?

Full = d². LoRA = 2·d·r.

Exercise

Why is the Bradley–Terry reward-model loss invariant to adding a constant to both $r_\theta(x, y_c)$ and $r_\theta(x, y_r)$?

Need a hint?

Look at what enters the sigmoid.

Exercise

Which combination correctly describes how DPO differs from classical RLHF with PPO?

Need a hint?

Think reward model, rollouts, and KL leash.

Exercise

A 13B-parameter model is currently served in FP16 (~26 GB). You quantize the weights to INT4. Roughly how much memory does the quantized model now use?

Need a hint?

INT4 = 0.5 bytes per parameter.

Exercise

In knowledge distillation, why does training the student on the teacher's full softened distribution beat training on hard labels alone?

Need a hint?

What does 'cat 78%, kitten 9%, dog 1.2%' tell you that 'cat' doesn't?

Exercise

A reward model turns pairwise human preferences into a scalar r(prompt, response) that later training maximizes. In the ML project loop, the reward model is which slot?

Need a hint?

It defines what 'good' means when there is no single correct label.

Exercise

Transfer test — a technique this course did NOT teach: an epsilon-greedy policy that occasionally takes a random action so an agent keeps discovering new states. Which slot of the ML project loop does exploration correspond to?

Need a hint?

It determines what experience — what data — the learner ever sees.