GRPO: Group Relative Policy OptimizationOpen in Colab

GRPO: Group Relative Policy Optimization

PPO-based RLHF needs a value network (critic) the same size as the policy, roughly doubling training memory, just to estimate how good a state is. GRPO (used to train DeepSeek-R1) removes the critic entirely: instead of learning a baseline, it samples a group of completions for the same prompt and uses the group's own reward statistics as the baseline. No value network, no GAE — just relative comparison within the group.

1: Sample a group of completions

For each prompt qq, sample GG completions from the current policy πθ\pi_\theta:

o1,o2,…,oG∼πθ(⋅∣q)o_1, o_2, \dots, o_G \sim \pi_\theta(\cdot \mid q)

Score each one with the reward function (a verifier for math/code, or a learned reward model): r1,r2,…,rGr_1, r_2, \dots, r_G.

2: Group-relative advantage

Normalize each reward against the group's own mean and standard deviation. This is the "group relative" part, and it replaces the critic:

A^i=ri−mean(r1,…,rG)std(r1,…,rG)+ϵ\hat A_i = \frac{r_i - \text{mean}(r_1, \dots, r_G)}{\text{std}(r_1, \dots, r_G) + \epsilon}

A completion is only "good" relative to its siblings answering the same prompt, which cancels out prompt-level difficulty — a hard prompt where every completion scores low still produces a useful learning signal, because the advantage is about relative rank, not absolute reward.

Why this replaces the critic

PPO's value network estimates V(q)V(q) so the advantage r−V(q)r - V(q) isolates how much better an action was than the expected outcome for that state. GRPO gets the same effect for free: mean(r1,…,rG)\text{mean}(r_1,\dots,r_G) is the empirical estimate of V(q)V(q), computed from the group instead of learned. It costs G×G\times more sampling per update but zero extra parameters or a second forward/backward pass through a critic network.

3: Clipped surrogate objective, applied per token

Every token in completion oio_i shares that completion's advantage A^i\hat A_i. Let ri,t(θ)=πθ(oi,t∣q,oi,<t)πθold(oi,t∣q,oi,<t)r_{i,t}(\theta) = \dfrac{\pi_\theta(o_{i,t} \mid q, o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t} \mid q, o_{i,<t})} be the per-token probability ratio against the policy that generated the group. GRPO clips it exactly like PPO:

J(θ)=Eq, {oi}[1G∑i=1G1∣oi∣∑t=1∣oi∣min⁡(ri,t(θ) A^i, clip(ri,t(θ),1−ϵ,1+ϵ) A^i)]J(\theta) = \mathbb{E}_{q,\,\{o_i\}}\left[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_i|}\sum_{t=1}^{|o_i|} \min\Big(r_{i,t}(\theta)\,\hat A_i,\ \text{clip}\big(r_{i,t}(\theta), 1-\epsilon, 1+\epsilon\big)\,\hat A_i\Big)\right]

The clip prevents any single update from moving the policy too far from what generated the group — without it, one lucky high-reward sample could push ri,tr_{i,t} to an extreme and destabilize training.

4: KL penalty against a frozen reference

To stop the policy drifting into reward hacking (e.g. degenerate outputs that fool the verifier), GRPO subtracts a KL term against a frozen reference policy πref\pi_{\text{ref}} (typically the SFT checkpoint before RL began):

LGRPO(θ)=−J(θ)+β Eq[DKL(πθ(⋅∣q) ∥ πref(⋅∣q))]\mathcal{L}_{\text{GRPO}}(\theta) = -J(\theta) + \beta\, \mathbb{E}_{q}\Big[D_{KL}\big(\pi_\theta(\cdot\mid q) \,\|\, \pi_{\text{ref}}(\cdot\mid q)\big)\Big]

β\beta trades off reward-seeking against staying close to a model known to produce coherent language.

Worked trace

Four sampled completions for one math prompt, verifier reward r∈{0,1}r \in \{0, 1\} (correct / incorrect):

completionreward rir_i
o1o_11
o2o_20
o3o_31
o4o_40

mean(r)=0.5\text{mean}(r) = 0.5, std(r)=0.5\text{std}(r) = 0.5. Advantages:

A^1=1−0.50.5=1,A^2=0−0.50.5=−1,A^3=1,A^4=−1\hat A_1 = \frac{1 - 0.5}{0.5} = 1, \quad \hat A_2 = \frac{0 - 0.5}{0.5} = -1, \quad \hat A_3 = 1, \quad \hat A_4 = -1

Every token in the two correct completions gets pushed up (positive advantage); every token in the two incorrect ones gets pushed down — even though the raw rewards were just 0/1, the group turned them into a signal with both a positive and a negative side, which is what makes the policy gradient informative.

GRPO vs PPO vs DPO

PPOGRPODPO
Baselinelearned value networkgroup mean rewardimplicit (via reference model in the loss)
Extra networkcritic (same size as policy)nonenone
Sampling per prompt1 completionGG completions1 pair (chosen/rejected)
Needs online rolloutsyesyesno (offline on a fixed preference dataset)

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

RelatedReferenced by
GRPO: Group Relative Policy Optim…Inverse Reinforcement Lea…From Policy Gradient to R…Reasoning ModelsPreference Tuning: RLHF &…