From Policy Gradient to RLHF (PPO)Open in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

From Policy Gradient to RLHF (PPO)

You just learned policy gradient: nudge the log-probability of an action by the advantage that followed. That's the right algorithm in principle but a terrible one in practice — the gradient is so noisy that one unlucky batch can launch the policy into a region it can't recover from. PPO fixes that with a single change: clip the update so any individual step is forced to stay close to the old policy. The same recipe (same clipped objective, same KL leash) is what turns a chat model into an aligned one. RLHF is just PPO with the LLM as the policy and a learned reward model as the environment.

Real-world examples: OpenAI's InstructGPT (the original "GPT-3 + RLHF" paper), ChatGPT-style alignment, Anthropic's Claude RLHF pipeline, Llama-2-Chat, and any modern frontier model that uses on-policy preference tuning.

Policy gradient: one-paragraph recap

REINFORCE estimates the gradient of expected return as

∇θJ(θ)=Eπθ ⁣[∇θlog⁡πθ(a∣s) A(s,a)],\nabla_\theta J(\theta) = \mathbb{E}_{\pi_\theta}\!\left[\nabla_\theta \log \pi_\theta(a\mid s)\, A(s,a)\right],

where A(s,a)=Gt−V(s)A(s,a) = G_t - V(s) is the advantage — how much better this action did than the value baseline. The signal is unbiased but extremely noisy: a single high-magnitude advantage can push some logit by a huge amount, and on the next sample the policy is somewhere completely different. Plain REINFORCE works on toy gridworlds and falls over on anything bigger. Actor-critic helps by replacing the raw return with a low-variance TD estimate, but the step size is still a free variable — nothing stops a single update from blowing up the policy.

Policy Gradient: Bandit

π(a) →true μA₀25.0%0.5A₁25.0%3.0A₂25.0%2.0A₃25.0%1.5Click Step to sample an action and observe the policy update…H(π) = 1.386
steps0
adv σ²0.00

The softmax policy π(a|θ) is updated each step via the policy gradient. Positive advantage increases the chosen action's probability; negative advantage decreases it. Toggle baseline to reduce variance.

PPO: clip the step, not the gradient

PPO (Schulman et al., 2017) doesn't change what the gradient is — it changes how big a step you're allowed to take. The trick is to keep a snapshot πθold\pi_{\theta_\text{old}} of the policy at the start of the update and write the objective in terms of an importance-sampling ratio:

ρt(θ)  =  πθ(at∣st)πθold(at∣st).\rho_t(\theta) \;=\; \frac{\pi_\theta(a_t \mid s_t)}{\pi_{\theta_\text{old}}(a_t \mid s_t)}.

At θ=θold\theta = \theta_\text{old}, ρt=1\rho_t = 1 and the gradient reduces to the standard policy gradient. As θ\theta drifts, ρt\rho_t measures how much more (or less) likely the action is under the new policy than under the snapshot. PPO maximises the clipped surrogate

LCLIP(θ)  =  Et ⁣[min⁡ ⁣(ρt(θ) At,  clip(ρt(θ), 1−ϵ, 1+ϵ) At)],\mathcal{L}^{\text{CLIP}}(\theta) \;=\; \mathbb{E}_t\!\left[\min\!\Big(\rho_t(\theta)\, A_t,\; \text{clip}\big(\rho_t(\theta),\, 1-\epsilon,\, 1+\epsilon\big)\, A_t\Big)\right],

with a typical ϵ=0.2\epsilon = 0.2. Read it slowly: for each token, take the smaller of (a) the un-clipped surrogate ρtAt\rho_t A_t and (b) the version where ρt\rho_t is hard-clipped to the band [1−ϵ,1+ϵ][1-\epsilon, 1+\epsilon].

What the clip is actually doing

Think of At>0A_t > 0 — a good action. The un-clipped term wants to push ρt\rho_t up without bound. The clipped term caps it at 1+ϵ1+\epsilon. The min picks the smaller of the two, so once ρt\rho_t exceeds 1+ϵ1+\epsilon the objective stops rewarding you for pushing it higher. The gradient on that token goes to zero. Symmetrically for At<0A_t < 0 and the lower bound. The net effect: one optimisation step can shift the policy by at most about ϵ\epsilon in importance ratio — a soft trust region that bounds KL drift without a hard constraint to solve.

This single change is the whole reason PPO replaced TRPO (which solved an explicit KL-constrained problem with a second-order method) as the default deep-RL workhorse. You get a trust-region-like update from plain Adam.

Reusing data, briefly

Because ρt\rho_t corrects for the difference between πθ\pi_\theta and πθold\pi_{\theta_\text{old}}, the same batch of rollouts can drive several gradient steps before the data goes too stale (typical: 3–10 epochs over a batch). Vanilla policy gradient throws each sample away after one update; PPO squeezes more out of every rollout while the clip keeps you honest.

RLHF = PPO on a language model

Now stretch one variable: let the policy be a language model πθ(yt∣x,y<t)\pi_\theta(y_t \mid x, y_{<t}). Then the RL setup is

RL conceptLanguage-model translation
State sts_tPrompt xx + tokens generated so far y<ty_{<t}
Action ata_tNext token yty_t
EpisodeOne full completion yy given prompt xx
RewardLearned reward model rϕ(x,y)r_\phi(x, y): scored on the final sequence
Reference policyThe SFT model πref\pi_\text{ref}

The classical RLHF objective is the same KL-regularised return you saw in the RLHF & DPO lesson:

LRLHF(θ)  =  Ex∼D, y∼πθ ⁣[rϕ(x,y)]  −  β⋅KL ⁣(πθ(⋅∣x) ∥ πref(⋅∣x)).\mathcal{L}_{\text{RLHF}}(\theta) \;=\; \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta}\!\big[r_\phi(x,y)\big] \;-\; \beta \cdot \mathrm{KL}\!\left(\pi_\theta(\cdot \mid x)\,\big\|\,\pi_\text{ref}(\cdot \mid x)\right).

The first term says "score high on the reward model". The second says "don't drift too far from the SFT model". In practice the KL term isn't computed once per completion — it's distributed across the sequence as a dense per-token reward. The per-token reward that PPO actually sees is

rt  =  rϕ(x,y)⋅1[t=T]⏟RM, only at end-of-sequence  −  β log⁡ ⁣πθ(yt∣⋅)πref(yt∣⋅)⏟KL penalty, every token.r_t \;=\; \underbrace{r_\phi(x, y) \cdot \mathbb{1}[t = T]}_{\text{RM, only at end-of-sequence}} \;-\; \underbrace{\beta\,\log\!\frac{\pi_\theta(y_t \mid \cdot)}{\pi_\text{ref}(y_t \mid \cdot)}}_{\text{KL penalty, every token}}.

So PPO only ever sees one combined reward signal. The end-of-sequence RM score is the sparse "did the whole completion turn out well?" component; the per-token log-ratio is the dense KL leash that keeps the policy on the manifold of fluent English.

One $\beta$, two failure modes

Set β\beta too low → the policy reward-hacks the RM: completions look like garbled token sequences that score absurdly high. Set β\beta too high → the policy barely moves off πref\pi_\text{ref} and the RM signal does nothing. The KL term is the only thing keeping RLHF on the rails; pick β\beta by watching held-out KL and generation quality, not just RM score.

The surface DPO collapses

The classical RLHF objective above (E[rϕ]−β KL(πθ∥πref)\mathbb{E}[r_\phi] - \beta\,\mathrm{KL}(\pi_\theta\|\pi_\text{ref})) has a known closed-form optimum, a Boltzmann reweighting of πref\pi_\text{ref} by the reward. DPO (covered in the fine-tuning lesson) reparameterises the reward as a log-ratio of policies and plugs it into the Bradley–Terry preference loss, and the entire PPO loop disappears, leaving a single classification objective. The clip-vs-collapse choice is the central design decision of modern preference tuning, and you now have both endpoints in view.

Exercise

In PPO's clipped surrogate $\mathcal{L}^{\text{CLIP}} = \mathbb{E}[\min(\rho_t A_t,\ \text{clip}(\rho_t, 1-\epsilon, 1+\epsilon)\,A_t)]$, what happens to the gradient on a token where $A_t > 0$ and $\rho_t$ has already grown past $1+\epsilon$?

Need a hint?

Walk through the `min` for a positive advantage when ρ_t is above the upper band.

Exercise

In a standard RLHF-with-PPO implementation, how is the reward model's score $r_\phi(x, y)$ and the KL leash $\beta\,\mathrm{KL}(\pi_\theta \| \pi_\text{ref})$ actually delivered to PPO as a per-token reward $r_t$?

Need a hint?

Which piece is sparse (one token) and which is dense (every token)?

Common mistakes

  1. Reading the clip as "clip the gradient". It's the objective that's clipped, not the gradient. The gradient simply becomes zero on tokens where the clip is binding, which is what kills the step there.
  2. Forgetting that the KL penalty is per token, not per sequence. Treating it as a single sequence-level scalar adds high-variance noise to the return; folding it into the per-token reward is what makes RLHF stable in practice.
  3. Treating the reward model as ground truth. rϕr_\phi is a learned approximation of human preference — push too hard and PPO finds adversarial completions the RM mis-scores. Always cap with the KL leash and a held-out human eval, not just RM score.
  4. Using a tiny ϵ\epsilon to "be safe". Too-small ϵ\epsilon kills every gradient and PPO stops learning. The standard ϵ=0.1\epsilon = 0.1–0.20.2 is small because the LLM has billions of parameters — for a small policy you'd use larger.

Key takeaways

  • PPO = policy gradient + clipped trust region. The ratio ρt\rho_t and clip at [1−ϵ,1+ϵ][1-\epsilon, 1+\epsilon] bound how far any single update can drift from πθold\pi_{\theta_\text{old}}.
  • The clip is on the objective, not the gradient — it just makes the gradient vanish past the bound, capping per-step KL drift without an explicit constraint.
  • RLHF = PPO on tokens with a reward model. State = prompt + prefix, action = next token, reward = RM at end-of-sequence + per-token KL penalty −βlog⁡(πθ/πref)-\beta \log(\pi_\theta/\pi_\text{ref}).
  • The classical RLHF objective E[rϕ]−β KL(πθ∥πref)\mathbb{E}[r_\phi] - \beta\,\mathrm{KL}(\pi_\theta\|\pi_\text{ref}) is exactly the surface that DPO rearranges away via a closed-form reward — same objective, no RL loop.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelatedGo deeper
From Policy Gradient to RLHF (PPO)Neural NetworksProbabilistic Models & EMPolicy Gradient & Actor-C…Preference Tuning: RLHF &…Reward ModelsGRPO: Group Relative Poli…Imitation LearningInverse Reinforcement Lea…