Preference Tuning: RLHF & DPOOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Preference Tuning: RLHF & DPO

The reward model from the last lesson is not the goal — it's a stepping stone. We still need a policy πθ\pi_\theta (the chat model) that actually produces high-reward responses at inference time. Preference tuning is the step that pushes the policy in that direction without letting it run off a cliff. Two recipes dominate in 2025: classical RLHF with PPO, and Direct Preference Optimization (DPO), which skips RL entirely and recovers the same objective as a single classification loss.

Real-world examples: the original InstructGPT and ChatGPT pipelines (SFT → RM → PPO); Anthropic's RLHF + Constitutional AI loops; LLaMA-2-Chat's two-RM PPO; Zephyr, Tulu-2, Mistral-Instruct, and most modern open chat models trained with DPO instead of PPO because it's an order of magnitude cheaper.

The RLHF pipeline

RLHF is three sequential stages that share one base model.

  1. SFT: fine-tune the pretrained model on a curated instruction dataset. This is the reference policy πref\pi_\text{ref} for everything that follows.
  2. Reward model: train rϕ(x,y)r_\phi(x, y) on chosen-vs-rejected pairs using the Bradley–Terry loss. Initialise from the SFT checkpoint.
  3. PPO: treat the LLM as an RL policy. State = prompt + generated prefix, action = next token, reward = rϕ(x,y)r_\phi(x, y) on the full completion. Optimise with policy gradient (PPO clip variant).

The RLHF objective adds a KL leash to the SFT model:

LRLHF(θ)=Ex∼D, y∼πθ[rϕ(x,y)]−β KL ⁣(πθ(⋅∣x) ∥ πref(⋅∣x)).\mathcal{L}_{\text{RLHF}}(\theta) = \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta}\big[r_\phi(x,y)\big] - \beta\, \mathrm{KL}\!\left(\pi_\theta(\cdot\mid x)\,\|\,\pi_{\text{ref}}(\cdot\mid x)\right).

Policy Gradient: Bandit

π(a) →true μA₀25.0%0.5A₁25.0%3.0A₂25.0%2.0A₃25.0%1.5Click Step to sample an action and observe the policy update…H(π) = 1.386
steps0
adv σ²0.00

The softmax policy π(a|θ) is updated each step via the policy gradient. Positive advantage increases the chosen action's probability; negative advantage decreases it. Toggle baseline to reduce variance.

The two terms pull in opposite directions:

  • The reward term E[rϕ]\mathbb{E}[r_\phi] pushes the policy toward responses the RM scores highly.
  • The KL term β⋅KL(πθ ∥ πref)\beta \cdot \mathrm{KL}(\pi_\theta \,\|\, \pi_\text{ref}) penalises drifting away from the SFT model. Without it, the policy will gleefully exploit every quirk of the RM (the reward-hacking trap from last lesson).

The hyperparameter β\beta is the KL budget. Set it too low and the policy reward-hacks until generations look nothing like English; set it too high and PPO barely moves off the SFT distribution.

Where the KL term actually lives

In implementation, the KL penalty is usually folded into the per-token reward as rt=rϕ⋅1[t=T]−βlog⁡πθ(yt∣⋅)πref(yt∣⋅)r_t = r_\phi \cdot \mathbb{1}[t = T] - \beta \log \frac{\pi_\theta(y_t \mid \cdot)}{\pi_\text{ref}(y_t \mid \cdot)}. The RM contributes a sparse reward at the final token; the KL contributes a dense per-token penalty. PPO sees one combined reward stream and doesn't need to know they came from different places.

PPO itself is the same clipped policy-gradient method from RL: importance-sampling ratio ρt=πθ(yt)/πθold(yt)\rho_t = \pi_\theta(y_t)/\pi_{\theta_\text{old}}(y_t), clipped to [1−ϵ,1+ϵ][1-\epsilon, 1+\epsilon], advantages from a value head. The only LLM-specific twist is "the reward arrives at end-of-sequence and the KL leash lives inside the reward."

Why a closed-form alternative exists

There's a textbook result for the KL-regularised RL objective above: the optimal policy is known in closed form as a Boltzmann-weighted version of the reference,

π∗(y∣x)  ∝  πref(y∣x) exp⁡ ⁣(r(x,y)β).\pi^*(y \mid x) \;\propto\; \pi_\text{ref}(y \mid x)\,\exp\!\left(\frac{r(x, y)}{\beta}\right).

That equation says: starting from the SFT model, reweight every possible response by how much reward it earns, with β\beta controlling the sharpness. High β\beta → mild reweighting, policy stays close to πref\pi_\text{ref}. Low β\beta → the reweighting dominates and π∗\pi^* concentrates on the highest-reward responses.

Rearranging gives the reward in terms of the optimal policy:

r(x,y)=βlog⁡π∗(y∣x)πref(y∣x)+Z(x),Z(x)=βlog⁡∑yπref(y∣x)exp⁡(r(x,y)/β).r(x, y) = \beta \log \frac{\pi^*(y \mid x)}{\pi_\text{ref}(y \mid x)} + Z(x), \qquad Z(x) = \beta \log \sum_y \pi_\text{ref}(y \mid x)\exp(r(x, y)/\beta).

The partition function Z(x)Z(x) only depends on the prompt — not on the response. This is the key observation that DPO leverages.

DPO: the closed-form loss

Plug the reward parameterisation r(x,y)=βlog⁡πθ(y∣x)πref(y∣x)+Z(x)r(x, y) = \beta \log \frac{\pi_\theta(y \mid x)}{\pi_\text{ref}(y \mid x)} + Z(x) into the Bradley–Terry preference probability:

P(yc≻yr∣x)=σ ⁣(r(x,yc)−r(x,yr)).P(y_c \succ y_r \mid x) = \sigma\!\big(r(x, y_c) - r(x, y_r)\big).

The Z(x)Z(x) terms cancel in the margin — the same partition function appears on both sides of the subtraction. What's left is a classification loss directly on the policy:

LDPO(θ)=− E(x,yc,yr)∼D ⁣[log⁡σ ⁣(βlog⁡πθ(yc∣x)πref(yc∣x)−βlog⁡πθ(yr∣x)πref(yr∣x))].\mathcal{L}_{\text{DPO}}(\theta) = -\,\mathbb{E}_{(x, y_c, y_r) \sim \mathcal{D}}\!\left[\log \sigma\!\left(\beta \log \frac{\pi_\theta(y_c \mid x)}{\pi_\text{ref}(y_c \mid x)} - \beta \log \frac{\pi_\theta(y_r \mid x)}{\pi_\text{ref}(y_r \mid x)}\right)\right].

That's the entire algorithm. No reward model. No PPO. No rollouts. No value head. Just two log-prob ratios per pair and a sigmoid.

What DPO is actually optimising

The bracket inside the sigmoid is an implicit reward margin: r^(x,y)=βlog⁡πθ(y∣x)πref(y∣x)\hat{r}(x, y) = \beta \log \frac{\pi_\theta(y \mid x)}{\pi_\text{ref}(y \mid x)} plays the role of the reward, and the gradient pushes r^(yc)−r^(yr)\hat{r}(y_c) - \hat{r}(y_r) upward on every preference pair. The reference policy plays the role of the KL leash automatically — divergence from πref\pi_\text{ref} shows up directly in the ratio. So DPO is preference tuning with the same KL-regularised objective as RLHF, just rearranged so there's no RL loop and no separate reward model to maintain.

The training loop is plain supervised learning:

  1. Sample a batch of preference pairs (x,yc,yr)(x, y_c, y_r).
  2. Run both πθ\pi_\theta and the frozen πref\pi_\text{ref} on each pair to get the four log-probs.
  3. Compute the loss, backprop, step.

The reference model is queried but never updated; it's typically the SFT checkpoint, cached on the same GPU.

RLHF vs DPO trade-offs

RLHF (PPO)DPO
PipelineSFT → RM → PPO (3 stages)SFT → DPO (2 stages)
Reward modelRequired (separate network)None (implicit via πθ/πref\pi_\theta / \pi_\text{ref})
On-policy samplesYes: rollouts during trainingNo: preference pairs are off-policy
MemoryPolicy + RM + value head + refPolicy + frozen ref
Hyperparametersβ\beta, PPO clip ϵ\epsilon, value-loss coef, GAE λ\lambda…β\beta
StabilityNotoriously fiddly; needs careful β\beta + clip tuningStable; closer to standard supervised fine-tuning
CeilingSometimes higher: can explore beyond preference dataBounded by the preference dataset's coverage
Compute4–10× a fine-tuning run≈ 1× a fine-tuning run

Every row in that table is easy to state and hard to feel. Walk the pipeline below on one prompt with nine candidate answers: fit a reward model on 240 sampled comparisons, watch the KL-regularised optimum form, then turn β\beta down until the policy collapses onto a response the reward model only ever extrapolated a score for. Then take the same comparisons down the DPO branch and watch it land on the identical formula with a different reward inside it. The annotation control is the one to spend time on — it changes only which responses got compared, and each round repairs exactly one of the two pipelines.

Preference tuning, step by step: RLHF and DPO on one prompt

Shared setupstep 1 of 8
SFT gives you a reference policy, not an aligned one

Nine possible answers to one debugging question. The SFT model spreads its mass over the safe middle — Names the cause and Generic advice take most of it — and almost never writes the best answer available.

Without this stage there is no π_ref at all, and both pipelines below need one: it is the distribution the KL term measures against and the model both are initialised from. But it is a starting point, not the goal — its expected quality is well short of what the response set allows.

Compare the two columns: the tallest reference bars are not the highest-quality rows.

KL budget βannotated
SFT referenceβ = 2 · pairs drawn from the SFT model · sample 1/6
Π_REFTRUE QUALITYAsks for more info30t13.0%1.0Generic advice70t26.0%3.0Names the cause100t31.0%5.5Cause + fix135t20.8%7.8Cause + fix + test170t8.5%8.2Minimal patch + repro145t0.4%9.8Padded restatement300t0.2%-1.3Correct but bloated380t0.1%0.2Wall of caveats460t<0.1%-7.7
π_ref

The SFT model scores 4.96 against a best available 9.83, and puts 0.4% on the best answer.

reference policy π_refheld-out true quality

Nine candidate answers, 240 annotator comparisons sampled from the SFT model, and two ways to turn them into a policy. Both branches end at the same closed form — π_ref reweighted by exp(r/β) — so the only difference is which reward goes in: two shared weights that generalise to unannotated text, or one free number per response that cannot. True quality is held out from both fits. The β and annotation controls re-run everything.

When does each win? DPO is the right default — same objective, far less infrastructure. RLHF/PPO still pays off when you have an excellent RM, plenty of compute, and want the policy to discover behaviours that aren't in the preference dataset (the on-policy rollouts let it explore). Most production open-weight chat models from 2024 onward are DPO; the largest closed-frontier models still use full RLHF (and beyond — rejection sampling, iterative DPO, RLAIF, online DPO variants).

The β knob is the same — but acts differently

In RLHF, β\beta scales an explicit KL penalty added to the reward. In DPO, β\beta scales the implicit reward margin inside the sigmoid: higher β\beta → sharper sigmoid → more weight on each preference pair → faster divergence from πref\pi_\text{ref}. Common DPO values are β∈[0.05,0.5]\beta \in [0.05, 0.5]. Set β\beta too high and the policy over-fits to the preference pairs and forgets general capability; too low and training barely moves.

Exercise

In the RLHF objective $\mathbb{E}[r_\phi(x,y)] - \beta \cdot \mathrm{KL}(\pi_\theta \| \pi_\text{ref})$, what does the KL term protect against?

Need a hint?

Without it, the policy is free to drift arbitrarily far from the SFT model.

Exercise

Which component does DPO eliminate compared to classical RLHF with PPO?

Need a hint?

DPO derives its loss from the closed-form optimum of the KL-regularised RL objective.

Exercise

The DPO loss is $-\log \sigma\!\left(\beta \log \frac{\pi_\theta(y_c|x)}{\pi_\text{ref}(y_c|x)} - \beta \log \frac{\pi_\theta(y_r|x)}{\pi_\text{ref}(y_r|x)}\right)$. Minimising it pushes the policy to do what?

Need a hint?

Which log-prob ratio should grow, which should shrink?

Common mistakes

  1. Skipping SFT. Both RLHF and DPO assume a competent reference model. Starting from a base model that hasn't seen instruction data first gives unstable training and weak final quality.
  2. Forgetting that DPO needs the reference policy at training time. You must hold πref\pi_\text{ref} in memory (or recompute it) for every batch — it appears in both the chosen and rejected log-prob ratios.
  3. Treating β\beta as a learning rate. It's a regularisation strength on divergence from πref\pi_\text{ref} — orthogonal to the optimiser's learning rate. Tune them separately.
  4. Pushing PPO until RM score saturates. Textbook reward hacking. Use the KL budget and held-out human eval as joint stopping criteria.
  5. Comparing DPO and PPO on RM score. They optimise different surfaces — RM score will favour PPO by construction. Compare on independent human eval or on a held-out preference dataset's pairwise accuracy.
  6. Using DPO without a strong base. DPO can only sharpen distinctions the model already roughly understands; if the preference signal demands behaviour the model literally can't generate, the loss can't conjure it.

Key takeaways

  • RLHF = SFT → RM → PPO with a KL leash to πref\pi_\text{ref}, so the policy doesn't reward-hack the RM. The objective is E[rϕ]−β⋅KL(πθ ∥ πref)\mathbb{E}[r_\phi] - \beta \cdot \mathrm{KL}(\pi_\theta \,\|\, \pi_\text{ref}).
  • The KL-regularised optimum has a closed form, π∗(y∣x)∝πref(y∣x)exp⁡(r(x,y)/β)\pi^*(y \mid x) \propto \pi_\text{ref}(y \mid x)\exp(r(x, y)/\beta). Inverting it shows the reward is the log-ratio βlog⁡(πθ/πref)\beta \log (\pi_\theta / \pi_\text{ref}) plus a prompt-only constant.
  • DPO plugs that closed-form reward into Bradley–Terry. The prompt-only constants cancel in the margin, leaving a plain −log⁡σ(⋅)-\log \sigma(\cdot) classification loss with no reward model and no RL loop.
  • DPO is cheaper, simpler, more stable, and has only one knob (β\beta). RLHF can reach higher ceilings via on-policy exploration but pays for it in compute and fiddliness. Modern production chat models lean DPO by default.

Related concepts

  • Reward Models: the scalar rϕr_\phi that PPO optimises against, and the Bradley–Terry loss that DPO inherits.
  • Policy Gradient: the underlying RL method PPO clips and extends; RLHF is policy-gradient on token sequences with a reward model.
  • Supervised Fine-Tuning: produces πref\pi_\text{ref}, the reference policy both pipelines hold the new model close to.
  • GRPO: Group Relative Policy Optimization: the critic-free RLHF objective used to train DeepSeek-R1

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelatedGo deeper
Preference Tuning: RLHF & DPOAttention & TransformersReinforcement LearningFoundation Models & Scali…From Policy Gradient to R…Reasoning ModelsPolicy Gradient & Actor-C…Reward ModelsSupervised Fine-Tuning & …GRPO: Group Relative Poli…