From Policy Gradient to RLHF (PPO)
You just learned policy gradient: nudge the log-probability of an action by the advantage that followed. That's the right algorithm in principle but a terrible one in practice — the gradient is so noisy that one unlucky batch can launch the policy into a region it can't recover from. PPO fixes that with a single change: clip the update so any individual step is forced to stay close to the old policy. The same recipe (same clipped objective, same KL leash) is what turns a chat model into an aligned one. RLHF is just PPO with the LLM as the policy and a learned reward model as the environment.
Real-world examples: OpenAI's InstructGPT (the original "GPT-3 + RLHF" paper), ChatGPT-style alignment, Anthropic's Claude RLHF pipeline, Llama-2-Chat, and any modern frontier model that uses on-policy preference tuning.
Policy gradient: one-paragraph recap
REINFORCE estimates the gradient of expected return as
where is the advantage — how much better this action did than the value baseline. The signal is unbiased but extremely noisy: a single high-magnitude advantage can push some logit by a huge amount, and on the next sample the policy is somewhere completely different. Plain REINFORCE works on toy gridworlds and falls over on anything bigger. Actor-critic helps by replacing the raw return with a low-variance TD estimate, but the step size is still a free variable — nothing stops a single update from blowing up the policy.
Policy Gradient: Bandit
The softmax policy π(a|θ) is updated each step via the policy gradient. Positive advantage increases the chosen action's probability; negative advantage decreases it. Toggle baseline to reduce variance.
PPO: clip the step, not the gradient
PPO (Schulman et al., 2017) doesn't change what the gradient is — it changes how big a step you're allowed to take. The trick is to keep a snapshot of the policy at the start of the update and write the objective in terms of an importance-sampling ratio:
At , and the gradient reduces to the standard policy gradient. As drifts, measures how much more (or less) likely the action is under the new policy than under the snapshot. PPO maximises the clipped surrogate
with a typical . Read it slowly: for each token, take the smaller of (a) the un-clipped surrogate and (b) the version where is hard-clipped to the band .
What the clip is actually doing
Think of — a good action. The un-clipped term wants to push up without bound. The clipped term caps it at . The min picks the smaller of the two, so once exceeds the objective stops rewarding you for pushing it higher. The gradient on that token goes to zero. Symmetrically for and the lower bound. The net effect: one optimisation step can shift the policy by at most about in importance ratio — a soft trust region that bounds KL drift without a hard constraint to solve.
This single change is the whole reason PPO replaced TRPO (which solved an explicit KL-constrained problem with a second-order method) as the default deep-RL workhorse. You get a trust-region-like update from plain Adam.
Reusing data, briefly
Because corrects for the difference between and , the same batch of rollouts can drive several gradient steps before the data goes too stale (typical: 3–10 epochs over a batch). Vanilla policy gradient throws each sample away after one update; PPO squeezes more out of every rollout while the clip keeps you honest.
RLHF = PPO on a language model
Now stretch one variable: let the policy be a language model . Then the RL setup is
| RL concept | Language-model translation |
|---|---|
| State | Prompt + tokens generated so far |
| Action | Next token |
| Episode | One full completion given prompt |
| Reward | Learned reward model : scored on the final sequence |
| Reference policy | The SFT model |
The classical RLHF objective is the same KL-regularised return you saw in the RLHF & DPO lesson:
The first term says "score high on the reward model". The second says "don't drift too far from the SFT model". In practice the KL term isn't computed once per completion — it's distributed across the sequence as a dense per-token reward. The per-token reward that PPO actually sees is
So PPO only ever sees one combined reward signal. The end-of-sequence RM score is the sparse "did the whole completion turn out well?" component; the per-token log-ratio is the dense KL leash that keeps the policy on the manifold of fluent English.
One $\beta$, two failure modes
Set too low → the policy reward-hacks the RM: completions look like garbled token sequences that score absurdly high. Set too high → the policy barely moves off and the RM signal does nothing. The KL term is the only thing keeping RLHF on the rails; pick by watching held-out KL and generation quality, not just RM score.
The surface DPO collapses
The classical RLHF objective above () has a known closed-form optimum, a Boltzmann reweighting of by the reward. DPO (covered in the fine-tuning lesson) reparameterises the reward as a log-ratio of policies and plugs it into the Bradley–Terry preference loss, and the entire PPO loop disappears, leaving a single classification objective. The clip-vs-collapse choice is the central design decision of modern preference tuning, and you now have both endpoints in view.
Exercise
In PPO's clipped surrogate $\mathcal{L}^{\text{CLIP}} = \mathbb{E}[\min(\rho_t A_t,\ \text{clip}(\rho_t, 1-\epsilon, 1+\epsilon)\,A_t)]$, what happens to the gradient on a token where $A_t > 0$ and $\rho_t$ has already grown past $1+\epsilon$?
Need a hint?
Walk through the `min` for a positive advantage when ρ_t is above the upper band.
Exercise
In a standard RLHF-with-PPO implementation, how is the reward model's score $r_\phi(x, y)$ and the KL leash $\beta\,\mathrm{KL}(\pi_\theta \| \pi_\text{ref})$ actually delivered to PPO as a per-token reward $r_t$?
Need a hint?
Which piece is sparse (one token) and which is dense (every token)?
Common mistakes
- Reading the clip as "clip the gradient". It's the objective that's clipped, not the gradient. The gradient simply becomes zero on tokens where the clip is binding, which is what kills the step there.
- Forgetting that the KL penalty is per token, not per sequence. Treating it as a single sequence-level scalar adds high-variance noise to the return; folding it into the per-token reward is what makes RLHF stable in practice.
- Treating the reward model as ground truth. is a learned approximation of human preference — push too hard and PPO finds adversarial completions the RM mis-scores. Always cap with the KL leash and a held-out human eval, not just RM score.
- Using a tiny to "be safe". Too-small kills every gradient and PPO stops learning. The standard – is small because the LLM has billions of parameters — for a small policy you'd use larger.
Key takeaways
- PPO = policy gradient + clipped trust region. The ratio and clip at bound how far any single update can drift from .
- The clip is on the objective, not the gradient — it just makes the gradient vanish past the bound, capping per-step KL drift without an explicit constraint.
- RLHF = PPO on tokens with a reward model. State = prompt + prefix, action = next token, reward = RM at end-of-sequence + per-token KL penalty .
- The classical RLHF objective is exactly the surface that DPO rearranges away via a closed-form reward — same objective, no RL loop.
Related concepts
- Policy Gradient & Actor-Critic: the on-policy estimator PPO clips.
- Preference Tuning: RLHF & DPO: the LLM-specific PPO pipeline and its closed-form alternative.
- Reward Models: the that PPO optimises against in RLHF.
- GRPO: Group Relative Policy Optimization: a critic-free alternative to PPO that normalizes rewards within a sampled group
- Imitation Learning: behavioral cloning, DAgger, and GAIL, the SFT-stage backdrop to RLHF
- Inverse Reinforcement Learning: recovering a reward from behavior, the classical root of reward modeling