Preference Tuning: RLHF & DPO
The reward model from the last lesson is not the goal — it's a stepping stone. We still need a policy (the chat model) that actually produces high-reward responses at inference time. Preference tuning is the step that pushes the policy in that direction without letting it run off a cliff. Two recipes dominate in 2025: classical RLHF with PPO, and Direct Preference Optimization (DPO), which skips RL entirely and recovers the same objective as a single classification loss.
Real-world examples: the original InstructGPT and ChatGPT pipelines (SFT → RM → PPO); Anthropic's RLHF + Constitutional AI loops; LLaMA-2-Chat's two-RM PPO; Zephyr, Tulu-2, Mistral-Instruct, and most modern open chat models trained with DPO instead of PPO because it's an order of magnitude cheaper.
The RLHF pipeline
RLHF is three sequential stages that share one base model.
- SFT: fine-tune the pretrained model on a curated instruction dataset. This is the reference policy for everything that follows.
- Reward model: train on chosen-vs-rejected pairs using the Bradley–Terry loss. Initialise from the SFT checkpoint.
- PPO: treat the LLM as an RL policy. State = prompt + generated prefix, action = next token, reward = on the full completion. Optimise with policy gradient (PPO clip variant).
The RLHF objective adds a KL leash to the SFT model:
Policy Gradient: Bandit
The softmax policy π(a|θ) is updated each step via the policy gradient. Positive advantage increases the chosen action's probability; negative advantage decreases it. Toggle baseline to reduce variance.
The two terms pull in opposite directions:
- The reward term pushes the policy toward responses the RM scores highly.
- The KL term penalises drifting away from the SFT model. Without it, the policy will gleefully exploit every quirk of the RM (the reward-hacking trap from last lesson).
The hyperparameter is the KL budget. Set it too low and the policy reward-hacks until generations look nothing like English; set it too high and PPO barely moves off the SFT distribution.
Where the KL term actually lives
In implementation, the KL penalty is usually folded into the per-token reward as . The RM contributes a sparse reward at the final token; the KL contributes a dense per-token penalty. PPO sees one combined reward stream and doesn't need to know they came from different places.
PPO itself is the same clipped policy-gradient method from RL: importance-sampling ratio , clipped to , advantages from a value head. The only LLM-specific twist is "the reward arrives at end-of-sequence and the KL leash lives inside the reward."
Why a closed-form alternative exists
There's a textbook result for the KL-regularised RL objective above: the optimal policy is known in closed form as a Boltzmann-weighted version of the reference,
That equation says: starting from the SFT model, reweight every possible response by how much reward it earns, with controlling the sharpness. High → mild reweighting, policy stays close to . Low → the reweighting dominates and concentrates on the highest-reward responses.
Rearranging gives the reward in terms of the optimal policy:
The partition function only depends on the prompt — not on the response. This is the key observation that DPO leverages.
DPO: the closed-form loss
Plug the reward parameterisation into the Bradley–Terry preference probability:
The terms cancel in the margin — the same partition function appears on both sides of the subtraction. What's left is a classification loss directly on the policy:
That's the entire algorithm. No reward model. No PPO. No rollouts. No value head. Just two log-prob ratios per pair and a sigmoid.
What DPO is actually optimising
The bracket inside the sigmoid is an implicit reward margin: plays the role of the reward, and the gradient pushes upward on every preference pair. The reference policy plays the role of the KL leash automatically — divergence from shows up directly in the ratio. So DPO is preference tuning with the same KL-regularised objective as RLHF, just rearranged so there's no RL loop and no separate reward model to maintain.
The training loop is plain supervised learning:
- Sample a batch of preference pairs .
- Run both and the frozen on each pair to get the four log-probs.
- Compute the loss, backprop, step.
The reference model is queried but never updated; it's typically the SFT checkpoint, cached on the same GPU.
RLHF vs DPO trade-offs
| RLHF (PPO) | DPO | |
|---|---|---|
| Pipeline | SFT → RM → PPO (3 stages) | SFT → DPO (2 stages) |
| Reward model | Required (separate network) | None (implicit via ) |
| On-policy samples | Yes: rollouts during training | No: preference pairs are off-policy |
| Memory | Policy + RM + value head + ref | Policy + frozen ref |
| Hyperparameters | , PPO clip , value-loss coef, GAE … | |
| Stability | Notoriously fiddly; needs careful + clip tuning | Stable; closer to standard supervised fine-tuning |
| Ceiling | Sometimes higher: can explore beyond preference data | Bounded by the preference dataset's coverage |
| Compute | 4–10× a fine-tuning run | ≈ 1× a fine-tuning run |
Every row in that table is easy to state and hard to feel. Walk the pipeline below on one prompt with nine candidate answers: fit a reward model on 240 sampled comparisons, watch the KL-regularised optimum form, then turn down until the policy collapses onto a response the reward model only ever extrapolated a score for. Then take the same comparisons down the DPO branch and watch it land on the identical formula with a different reward inside it. The annotation control is the one to spend time on — it changes only which responses got compared, and each round repairs exactly one of the two pipelines.
Preference tuning, step by step: RLHF and DPO on one prompt
SFT gives you a reference policy, not an aligned one
Nine possible answers to one debugging question. The SFT model spreads its mass over the safe middle — Names the cause and Generic advice take most of it — and almost never writes the best answer available.
Without this stage there is no π_ref at all, and both pipelines below need one: it is the distribution the KL term measures against and the model both are initialised from. But it is a starting point, not the goal — its expected quality is well short of what the response set allows.
Compare the two columns: the tallest reference bars are not the highest-quality rows.
π_ref
The SFT model scores 4.96 against a best available 9.83, and puts 0.4% on the best answer.
Nine candidate answers, 240 annotator comparisons sampled from the SFT model, and two ways to turn them into a policy. Both branches end at the same closed form — π_ref reweighted by exp(r/β) — so the only difference is which reward goes in: two shared weights that generalise to unannotated text, or one free number per response that cannot. True quality is held out from both fits. The β and annotation controls re-run everything.
When does each win? DPO is the right default — same objective, far less infrastructure. RLHF/PPO still pays off when you have an excellent RM, plenty of compute, and want the policy to discover behaviours that aren't in the preference dataset (the on-policy rollouts let it explore). Most production open-weight chat models from 2024 onward are DPO; the largest closed-frontier models still use full RLHF (and beyond — rejection sampling, iterative DPO, RLAIF, online DPO variants).
The β knob is the same — but acts differently
In RLHF, scales an explicit KL penalty added to the reward. In DPO, scales the implicit reward margin inside the sigmoid: higher → sharper sigmoid → more weight on each preference pair → faster divergence from . Common DPO values are . Set too high and the policy over-fits to the preference pairs and forgets general capability; too low and training barely moves.
Exercise
In the RLHF objective $\mathbb{E}[r_\phi(x,y)] - \beta \cdot \mathrm{KL}(\pi_\theta \| \pi_\text{ref})$, what does the KL term protect against?
Need a hint?
Without it, the policy is free to drift arbitrarily far from the SFT model.
Exercise
Which component does DPO eliminate compared to classical RLHF with PPO?
Need a hint?
DPO derives its loss from the closed-form optimum of the KL-regularised RL objective.
Exercise
The DPO loss is $-\log \sigma\!\left(\beta \log \frac{\pi_\theta(y_c|x)}{\pi_\text{ref}(y_c|x)} - \beta \log \frac{\pi_\theta(y_r|x)}{\pi_\text{ref}(y_r|x)}\right)$. Minimising it pushes the policy to do what?
Need a hint?
Which log-prob ratio should grow, which should shrink?
Common mistakes
- Skipping SFT. Both RLHF and DPO assume a competent reference model. Starting from a base model that hasn't seen instruction data first gives unstable training and weak final quality.
- Forgetting that DPO needs the reference policy at training time. You must hold in memory (or recompute it) for every batch — it appears in both the chosen and rejected log-prob ratios.
- Treating as a learning rate. It's a regularisation strength on divergence from — orthogonal to the optimiser's learning rate. Tune them separately.
- Pushing PPO until RM score saturates. Textbook reward hacking. Use the KL budget and held-out human eval as joint stopping criteria.
- Comparing DPO and PPO on RM score. They optimise different surfaces — RM score will favour PPO by construction. Compare on independent human eval or on a held-out preference dataset's pairwise accuracy.
- Using DPO without a strong base. DPO can only sharpen distinctions the model already roughly understands; if the preference signal demands behaviour the model literally can't generate, the loss can't conjure it.
Key takeaways
- RLHF = SFT → RM → PPO with a KL leash to , so the policy doesn't reward-hack the RM. The objective is .
- The KL-regularised optimum has a closed form, . Inverting it shows the reward is the log-ratio plus a prompt-only constant.
- DPO plugs that closed-form reward into Bradley–Terry. The prompt-only constants cancel in the margin, leaving a plain classification loss with no reward model and no RL loop.
- DPO is cheaper, simpler, more stable, and has only one knob (). RLHF can reach higher ceilings via on-policy exploration but pays for it in compute and fiddliness. Modern production chat models lean DPO by default.
Related concepts
- Reward Models: the scalar that PPO optimises against, and the Bradley–Terry loss that DPO inherits.
- Policy Gradient: the underlying RL method PPO clips and extends; RLHF is policy-gradient on token sequences with a reward model.
- Supervised Fine-Tuning: produces , the reference policy both pipelines hold the new model close to.
- GRPO: Group Relative Policy Optimization: the critic-free RLHF objective used to train DeepSeek-R1