Policy Gradient & Actor-Critic
DQN learns values and acts greedily on them. Policy gradient methods skip the middleman and optimize the policy directly by gradient ascent on expected return. This unlocks continuous actions, naturally stochastic policies, and is the backbone of modern RL — PPO, the algorithm behind RLHF for language models, is a policy-gradient method.
Real-world examples: Robotics with continuous control, AlphaGo's policy network, RLHF for ChatGPT/Claude, continuous control benchmarks (MuJoCo), game playing.
Value-based vs policy-based
| Value-based (DQN) | Policy-based (PG) | |
|---|---|---|
| Learns | , acts greedily | directly |
| Action space | Discrete only | Discrete or continuous |
| Policy | Deterministic (+ ε) | Naturally stochastic |
| Convergence | Can oscillate | Smoother, but high variance |
Intuition
Value methods ask "how good is each action?" then pick the best. Policy methods ask "let me adjust my action probabilities to make good outcomes more likely." If an episode went well, push up the probability of the actions you took; if it went badly, push them down. That's the whole idea.
The policy gradient theorem
We want to maximize expected return . The remarkable result is that its gradient has a form we can estimate from samples:
Read it as: increase the log-probability of actions, weighted by the return that followed them. Good returns ( large) pull those actions' probabilities up; bad returns push them down. No model of the environment, no value table required.
REINFORCE: the basic algorithm
The simplest policy-gradient method (Williams, 1992):
- Run a full episode with the current policy .
- Compute the return from each step to the end.
- Update .
import torch, torch.nn as nn
policy = nn.Sequential(nn.Linear(obs_dim, 128), nn.ReLU(),
nn.Linear(128, n_actions)) # logits
def reinforce_update(log_probs, rewards, optim, gamma=0.99):
# discounted returns, computed backward
returns, G = [], 0
for r in reversed(rewards):
G = r + gamma * G
returns.insert(0, G)
returns = torch.tensor(returns)
returns = (returns - returns.mean()) / (returns.std() + 1e-8) # baseline+norm
loss = -(torch.stack(log_probs) * returns).sum() # ascent → minimize -J
optim.zero_grad(); loss.backward(); optim.step()
return loss.item()
Why the minus sign?
Optimizers minimize. To do gradient ascent on , we minimize . Hence the negative log-prob times return — maximizing return becomes minimizing its negative.
The variance problem
REINFORCE works but learns slowly: the return is a noisy Monte-Carlo estimate over a whole episode, so the gradient has high variance. Two standard cures:
- Baseline subtraction — subtract a state-dependent baseline from the return. It doesn't bias the gradient but slashes variance:
The best baseline is the value function , which gives the advantage : "how much better than average was this action?"
- Bootstrapping: replace the full-episode return with a one-step estimate, just like TD learning.
Combine both and you get actor-critic.
Policy Gradient: Bandit
The softmax policy π(a|θ) is updated each step via the policy gradient. Positive advantage increases the chosen action's probability; negative advantage decreases it. Toggle baseline to reduce variance.
Watch the baseline work
REINFORCE — what a baseline does, and what it must not depend on
A one-step episode: three actions, a softmax policy π = (0.309, 0.461, 0.229), and rewards drawn around means (10.0, 10.5, 9.5) with σ = 1. One step is the simplest case where REINFORCE is still REINFORCE, and it buys something worth having — **every quantity below is available in closed form**, so nothing is estimated that could have been derived. The exact policy gradient is ∇J = Σ_a π_a·E[G|a]·(e_a − π) = (-0.0359, 0.1771, -0.1412), and the value of the current policy is V = E[G] = 10.1162.
A one-step episode with a softmax policy, chosen so the exact gradient, the exact variance under any baseline, and the exact variance-minimising baseline are all available in closed form. The lesson says a baseline does not bias the gradient but slashes variance: measured, four baselines including two deliberately wrong ones all recover the exact gradient, while variance runs 64.6 with none against 0.70 at V — a 93x reduction. The second payoff qualifies 'the best baseline is V(s)': the true optimum is a gradient-weighted average of returns, not the mean, but it is only 1.3% better than V. The mistake that does matter is a baseline depending on the action, which drives the expected gradient to exactly zero and stops learning with no error at all.
What to notice:
- Frame 2: where the variance comes from. A single episode returns , so the update scales the score by ten, even though the informative part — how much better than average that action is — is only 0.38. The absolute level of carries none of the signal and all of the variance.
- Frame 5: "doesn't bias the gradient" does not depend on choosing well. Four baselines, including two deliberately wrong ones, all recover the exact gradient; only the variance changes (64.5 → 0.70 at , 1.49 one unit away). Unbiasedness is free because identically, so for any action-independent .
- Frame 6: is not actually the optimal baseline. The variance-minimizing choice is , a gradient-weighted average of returns rather than the plain mean. It beats by only 1.3%, though, against the 93× that already bought, which is why every textbook says and is right to.
The failure that does matter is different: a baseline depending on the action rather than the state. Setting drives the expected gradient to essentially zero (norm against a true 0.229) — learning stops dead with no error raised. That is why the baseline above is written and not , and why the actor-critic advantage below stays legitimate: is a function of the state alone.
Actor-Critic
Two cooperating networks:
- Actor — chooses actions (the policy).
- Critic — estimates state value, providing the baseline/advantage.
The actor updates using the advantage from the critic; the critic learns by TD just like before:
Actor and critic, in plain terms
The actor acts. The critic judges. The critic's TD error tells the actor "that action did better/worse than I expected from this state" — a low-variance learning signal that arrives every step instead of only at episode end. The actor adjusts; the critic sharpens its judgment. They improve together.
Worked example: advantage sign
State value estimate . The agent takes an action, gets , lands in with , .
Advantage . Positive → the action did better than the critic expected → increase its probability. A negative advantage would decrease it.
Modern policy-gradient methods
| Method | Idea |
|---|---|
| A2C / A3C | Parallel actor-critic, synchronous/asynchronous |
| TRPO | Limit policy change with a trust region |
| PPO | Clipped objective: simple, robust, the default today |
| SAC / DDPG | Off-policy actor-critic for continuous control |
PPO (Proximal Policy Optimization) is the workhorse: it clips the policy update so each step can't move too far, giving stable training with little tuning. It's the RL algorithm used in RLHF to align large language models.
On-policy means sample-hungry
Vanilla policy gradient and PPO are on-policy — each batch of data must come from the current policy, then is thrown away. That makes them more sample-expensive than off-policy DQN. The payoff is stability and continuous-action support. Use a simulator or lots of parallel actors.
Exercise
The policy gradient ∇J = E[∇log π(a|s) · G] adjusts action probabilities how?
Need a hint?
What is the weight on each action's log-probability?
Exercise
Why subtract a baseline (e.g. the value function) from the return in policy gradient methods?
Need a hint?
Does it change the expected gradient? What about its variance?
Exercise
In actor-critic methods, what are the roles of the actor and the critic?
Need a hint?
One chooses actions, one evaluates them.
Common mistakes
- No baseline: plain REINFORCE has crippling variance; always subtract a baseline / use a critic.
- Reusing old data on-policy: policy gradient data is stale after one update (PPO's clip lets you reuse it a few times, not indefinitely).
- Forgetting entropy regularization: without it the policy collapses to deterministic too early and stops exploring.
Key takeaways
- Policy gradient optimizes directly — supporting continuous, stochastic actions.
- It raises the log-probability of actions weighted by their return.
- A baseline/advantage cuts variance; actor-critic combines policy + value nets.
- PPO is the robust default and powers RLHF for language models.
Related concepts
- Deep Q-Networks: the value-based alternative
- Markov Decision Processes: the objective being optimized
- Gradient Descent: the ascent that updates the policy
- Multi-Agent Reinforcement Learning: Markov games, non-stationarity, and CTDE actor-critic methods
- Imitation Learning: GAIL uses policy gradients to match an expert's occupancy measure