GRPO: Group Relative Policy Optimization
PPO-based RLHF needs a value network (critic) the same size as the policy, roughly doubling training memory, just to estimate how good a state is. GRPO (used to train DeepSeek-R1) removes the critic entirely: instead of learning a baseline, it samples a group of completions for the same prompt and uses the group's own reward statistics as the baseline. No value network, no GAE — just relative comparison within the group.
1: Sample a group of completions
For each prompt , sample completions from the current policy :
Score each one with the reward function (a verifier for math/code, or a learned reward model): .
2: Group-relative advantage
Normalize each reward against the group's own mean and standard deviation. This is the "group relative" part, and it replaces the critic:
A completion is only "good" relative to its siblings answering the same prompt, which cancels out prompt-level difficulty — a hard prompt where every completion scores low still produces a useful learning signal, because the advantage is about relative rank, not absolute reward.
Why this replaces the critic
PPO's value network estimates so the advantage isolates how much better an action was than the expected outcome for that state. GRPO gets the same effect for free: is the empirical estimate of , computed from the group instead of learned. It costs more sampling per update but zero extra parameters or a second forward/backward pass through a critic network.
3: Clipped surrogate objective, applied per token
Every token in completion shares that completion's advantage . Let be the per-token probability ratio against the policy that generated the group. GRPO clips it exactly like PPO:
The clip prevents any single update from moving the policy too far from what generated the group — without it, one lucky high-reward sample could push to an extreme and destabilize training.
4: KL penalty against a frozen reference
To stop the policy drifting into reward hacking (e.g. degenerate outputs that fool the verifier), GRPO subtracts a KL term against a frozen reference policy (typically the SFT checkpoint before RL began):
trades off reward-seeking against staying close to a model known to produce coherent language.
Worked trace
Four sampled completions for one math prompt, verifier reward (correct / incorrect):
| completion | reward |
|---|---|
| 1 | |
| 0 | |
| 1 | |
| 0 |
, . Advantages:
Every token in the two correct completions gets pushed up (positive advantage); every token in the two incorrect ones gets pushed down — even though the raw rewards were just 0/1, the group turned them into a signal with both a positive and a negative side, which is what makes the policy gradient informative.
GRPO vs PPO vs DPO
| PPO | GRPO | DPO | |
|---|---|---|---|
| Baseline | learned value network | group mean reward | implicit (via reference model in the loss) |
| Extra network | critic (same size as policy) | none | none |
| Sampling per prompt | 1 completion | completions | 1 pair (chosen/rejected) |
| Needs online rollouts | yes | yes | no (offline on a fixed preference dataset) |
Related concepts
- Reasoning Models: where GRPO is used to train DeepSeek-R1-style chain-of-thought
- Preference Tuning: RLHF & DPO: the value-network-based alternative GRPO simplifies
- From Policy Gradient to RLHF: the PPO clip objective GRPO reuses per-token