Policy Gradient & Actor-CriticOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Policy Gradient & Actor-Critic

DQN learns values and acts greedily on them. Policy gradient methods skip the middleman and optimize the policy πθ(a∣s)\pi_\theta(a\mid s) directly by gradient ascent on expected return. This unlocks continuous actions, naturally stochastic policies, and is the backbone of modern RL — PPO, the algorithm behind RLHF for language models, is a policy-gradient method.

Real-world examples: Robotics with continuous control, AlphaGo's policy network, RLHF for ChatGPT/Claude, continuous control benchmarks (MuJoCo), game playing.

Value-based vs policy-based

Value-based (DQN)Policy-based (PG)
LearnsQ(s,a)Q(s,a), acts greedilyπθ(a∣s)\pi_\theta(a\mid s) directly
Action spaceDiscrete onlyDiscrete or continuous
PolicyDeterministic (+ ε)Naturally stochastic
ConvergenceCan oscillateSmoother, but high variance

Intuition

Value methods ask "how good is each action?" then pick the best. Policy methods ask "let me adjust my action probabilities to make good outcomes more likely." If an episode went well, push up the probability of the actions you took; if it went badly, push them down. That's the whole idea.

The policy gradient theorem

We want to maximize expected return J(θ)=Eπθ[G]J(\theta) = \mathbb{E}_{\pi_\theta}[G]. The remarkable result is that its gradient has a form we can estimate from samples:

∇θJ(θ)=Eπθ[∇θlog⁡πθ(a∣s)  Gt]\nabla_\theta J(\theta) = \mathbb{E}_{\pi_\theta}\Big[\nabla_\theta \log \pi_\theta(a\mid s)\; G_t\Big]

Read it as: increase the log-probability of actions, weighted by the return that followed them. Good returns (GtG_t large) pull those actions' probabilities up; bad returns push them down. No model of the environment, no value table required.

REINFORCE: the basic algorithm

The simplest policy-gradient method (Williams, 1992):

  1. Run a full episode with the current policy πθ\pi_\theta.
  2. Compute the return GtG_t from each step to the end.
  3. Update θ←θ+α∑t∇θlog⁡πθ(at∣st) Gt\theta \leftarrow \theta + \alpha \sum_t \nabla_\theta \log \pi_\theta(a_t\mid s_t)\, G_t.
Python
import torch, torch.nn as nn

policy = nn.Sequential(nn.Linear(obs_dim, 128), nn.ReLU(),
                       nn.Linear(128, n_actions))   # logits

def reinforce_update(log_probs, rewards, optim, gamma=0.99):
    # discounted returns, computed backward
    returns, G = [], 0
    for r in reversed(rewards):
        G = r + gamma * G
        returns.insert(0, G)
    returns = torch.tensor(returns)
    returns = (returns - returns.mean()) / (returns.std() + 1e-8)  # baseline+norm

    loss = -(torch.stack(log_probs) * returns).sum()   # ascent → minimize -J
    optim.zero_grad(); loss.backward(); optim.step()
    return loss.item()

Why the minus sign?

Optimizers minimize. To do gradient ascent on JJ, we minimize −J-J. Hence the negative log-prob times return — maximizing return becomes minimizing its negative.

The variance problem

REINFORCE works but learns slowly: the return GtG_t is a noisy Monte-Carlo estimate over a whole episode, so the gradient has high variance. Two standard cures:

  1. Baseline subtraction — subtract a state-dependent baseline b(s)b(s) from the return. It doesn't bias the gradient but slashes variance:
∇θJ=E[∇θlog⁡πθ(a∣s) (Gt−b(s))]\nabla_\theta J = \mathbb{E}\big[\nabla_\theta \log \pi_\theta(a\mid s)\,(G_t - b(s))\big]

The best baseline is the value function V(s)V(s), which gives the advantage A(s,a)=Gt−V(s)A(s,a) = G_t - V(s): "how much better than average was this action?"

  1. Bootstrapping: replace the full-episode return with a one-step estimate, just like TD learning.

Combine both and you get actor-critic.

Policy Gradient: Bandit

π(a) →true μA₀25.0%0.5A₁25.0%3.0A₂25.0%2.0A₃25.0%1.5Click Step to sample an action and observe the policy update…H(π) = 1.386
steps0
adv σ²0.00

The softmax policy π(a|θ) is updated each step via the policy gradient. Positive advantage increases the chosen action's probability; negative advantage decreases it. Toggle baseline to reduce variance.

Watch the baseline work

REINFORCE — what a baseline does, and what it must not depend on

1/6

A one-step episode: three actions, a softmax policy π = (0.309, 0.461, 0.229), and rewards drawn around means (10.0, 10.5, 9.5) with σ = 1. One step is the simplest case where REINFORCE is still REINFORCE, and it buys something worth having — **every quantity below is available in closed form**, so nothing is estimated that could have been derived. The exact policy gradient is ∇J = Σ_a π_a·E[G|a]·(e_a − π) = (-0.0359, 0.1771, -0.1412), and the value of the current policy is V = E[G] = 10.1162.

python
1for episode in range(N):
2 a = sample(pi(theta))
3 G = play(a) # the return
4
5 # score function of the chosen action
6 g = onehot(a) - pi(theta)
7
8 # baseline must not depend on a
9 adv = G - b
10
11 theta += alpha * g * adv
state
policy π(a) and mean reward per action
π(a0)
0.309
E[G|a0]
10.0
π(a1)
0.461
E[G|a1]
10.5
π(a2)
0.229
E[G|a2]
9.5
exact quantities
∇J (exact) → (-0.0359, 0.1771, -0.1412)V = E[G] → 10.1162best action → a1spread of means → 1.0

A one-step episode with a softmax policy, chosen so the exact gradient, the exact variance under any baseline, and the exact variance-minimising baseline are all available in closed form. The lesson says a baseline does not bias the gradient but slashes variance: measured, four baselines including two deliberately wrong ones all recover the exact gradient, while variance runs 64.6 with none against 0.70 at V — a 93x reduction. The second payoff qualifies 'the best baseline is V(s)': the true optimum is a gradient-weighted average of returns, not the mean, but it is only 1.3% better than V. The mistake that does matter is a baseline depending on the action, which drives the expected gradient to exactly zero and stops learning with no error at all.

What to notice:

  • Frame 2: where the variance comes from. A single episode returns G≈10.5G \approx 10.5, so the update scales the score by ten, even though the informative part — how much better than average that action is — is only 0.38. The absolute level of GG carries none of the signal and all of the variance.
  • Frame 5: "doesn't bias the gradient" does not depend on choosing well. Four baselines, including two deliberately wrong ones, all recover the exact gradient; only the variance changes (64.5 → 0.70 at VV, 1.49 one unit away). Unbiasedness is free because E[∇log⁡π]=0\mathbb{E}[\nabla\log\pi] = 0 identically, so E[∇log⁡π⋅b]=0\mathbb{E}[\nabla\log\pi \cdot b] = 0 for any action-independent bb.
  • Frame 6: VV is not actually the optimal baseline. The variance-minimizing choice is b⋆=E[∥g∥2G]/E[∥g∥2]b^\star = \mathbb{E}[\lVert g\rVert^2 G]/\mathbb{E}[\lVert g\rVert^2], a gradient-weighted average of returns rather than the plain mean. It beats VV by only 1.3%, though, against the 93× that VV already bought, which is why every textbook says VV and is right to.

The failure that does matter is different: a baseline depending on the action rather than the state. Setting b(a)=E[G∣a]b(a) = \mathbb{E}[G \mid a] drives the expected gradient to essentially zero (norm 1.8×10−31.8\times10^{-3} against a true 0.229) — learning stops dead with no error raised. That is why the baseline above is written b(s)b(s) and not b(s,a)b(s,a), and why the actor-critic advantage below stays legitimate: VV is a function of the state alone.

Actor-Critic

Two cooperating networks:

  • Actor πθ(a∣s)\pi_\theta(a\mid s) — chooses actions (the policy).
  • Critic Vϕ(s)V_\phi(s) — estimates state value, providing the baseline/advantage.

The actor updates using the advantage from the critic; the critic learns VV by TD just like before:

At=rt+γVϕ(st+1)−Vϕ(st)(TD error = advantage estimate)A_t = r_t + \gamma V_\phi(s_{t+1}) - V_\phi(s_t) \quad(\text{TD error = advantage estimate}) θ←θ+α ∇θlog⁡πθ(at∣st) Atϕ←ϕ+β At∇ϕVϕ(st)\theta \leftarrow \theta + \alpha\,\nabla_\theta \log\pi_\theta(a_t\mid s_t)\,A_t \qquad \phi \leftarrow \phi + \beta\,A_t \nabla_\phi V_\phi(s_t)

Actor and critic, in plain terms

The actor acts. The critic judges. The critic's TD error tells the actor "that action did better/worse than I expected from this state" — a low-variance learning signal that arrives every step instead of only at episode end. The actor adjusts; the critic sharpens its judgment. They improve together.

Worked example: advantage sign

State value estimate V(s)=8V(s) = 8. The agent takes an action, gets r=1r = 1, lands in s′s' with V(s′)=10V(s') = 10, γ=0.9\gamma = 0.9.

Advantage A=1+0.9⋅10−8=1+9−8=+2A = 1 + 0.9\cdot 10 - 8 = 1 + 9 - 8 = +2. Positive → the action did better than the critic expected → increase its probability. A negative advantage would decrease it.

Modern policy-gradient methods

MethodIdea
A2C / A3CParallel actor-critic, synchronous/asynchronous
TRPOLimit policy change with a trust region
PPOClipped objective: simple, robust, the default today
SAC / DDPGOff-policy actor-critic for continuous control

PPO (Proximal Policy Optimization) is the workhorse: it clips the policy update so each step can't move too far, giving stable training with little tuning. It's the RL algorithm used in RLHF to align large language models.

On-policy means sample-hungry

Vanilla policy gradient and PPO are on-policy — each batch of data must come from the current policy, then is thrown away. That makes them more sample-expensive than off-policy DQN. The payoff is stability and continuous-action support. Use a simulator or lots of parallel actors.

Exercise

The policy gradient ∇J = E[∇log π(a|s) · G] adjusts action probabilities how?

Need a hint?

What is the weight on each action's log-probability?

Exercise

Why subtract a baseline (e.g. the value function) from the return in policy gradient methods?

Need a hint?

Does it change the expected gradient? What about its variance?

Exercise

In actor-critic methods, what are the roles of the actor and the critic?

Need a hint?

One chooses actions, one evaluates them.

Common mistakes

  1. No baseline: plain REINFORCE has crippling variance; always subtract a baseline / use a critic.
  2. Reusing old data on-policy: policy gradient data is stale after one update (PPO's clip lets you reuse it a few times, not indefinitely).
  3. Forgetting entropy regularization: without it the policy collapses to deterministic too early and stops exploring.

Key takeaways

  • Policy gradient optimizes πθ\pi_\theta directly — supporting continuous, stochastic actions.
  • It raises the log-probability of actions weighted by their return.
  • A baseline/advantage cuts variance; actor-critic combines policy + value nets.
  • PPO is the robust default and powers RLHF for language models.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelatedGo deeper
Policy Gradient & Actor-CriticNeural NetworksProbabilistic Models & EMMarkov Decision ProcessesDeep Q-NetworksGradient DescentFrom Policy Gradient to R…Preference Tuning: RLHF &…Multi-Agent Reinforcement…Imitation Learning