Deep Q-Networks
Tabular Q-learning dies when the state space is huge — you cannot tabulate every Atari screen. A Deep Q-Network (DQN) replaces the Q-table with a neural network that generalizes across states. DeepMind's 2015 DQN learned to play 49 Atari games from raw pixels at human level using one architecture — the result that launched modern deep RL.
Real-world examples: Atari from pixels, game AI, robotics control, resource scheduling, the foundation for Rainbow, Double/Dueling DQN, and beyond.
From table to function approximator
Instead of storing in a grid, a network takes the state as input and outputs a Q-value for every action at once:
For Atari, is a stack of 4 game frames, the network is a CNN, and the output is one Q-value per joystick action. Now nearby states share parameters, so the agent generalizes to screens it has never seen — impossible with a table.
Intuition
A table memorizes; a network generalizes. Two slightly different game screens produce similar features and similar Q-values automatically — the network interpolates. That's the only way to handle state spaces too large to ever visit fully.
Training as regression
We turn the Q-learning update into a supervised loss. The target is the same Bellman backup; we minimize the squared TD error:
and update by gradient descent. But done naively this is unstable — for two deep reasons, each with a fix.
Problem 1: correlated data → Experience Replay
Consecutive transitions in an episode are highly correlated, and SGD assumes i.i.d. samples. DQN stores transitions in a replay buffer and trains on random minibatches drawn from it.
from collections import deque
import random
replay = deque(maxlen=100_000)
replay.append((s, a, r, s2, done)) # store every step
batch = random.sample(replay, 32) # train on a random, decorrelated batch
Benefits: breaks temporal correlation, and reuses each experience many times (sample-efficient).
Same data, same updates, different order
Raise the learning rate and the gap widens — 3.33× at 0.02, 11.89× at 0.05 — because a larger step lets each stretch of near-identical samples drag the weights further before the next region arrives. That is why “just use a smaller learning rate” is not the fix: it trades the problem for slowness. Replay changes the order, and the order is the problem.
Weight-space trajectories of the two fits. The agent walks slowly through feature space so consecutive samples are near-duplicates; the sequential run (rose) is dragged around by whichever region it is currently in, while the shuffled run (teal) sees a fresh mix at every step. The white cross is the true weight vector. Averages are over 400 independent trajectories.
"SGD assumes i.i.d. samples" is easy to nod along to and hard to feel. Here the same data is fitted twice — same learning rate, same number of updates, differing only in the order:
| learning rate | sequential | shuffled | ratio |
|---|---|---|---|
| 0.02 | 0.2575 | 0.0774 | 3.33× |
| 0.05 | 0.1928 | 0.0162 | 11.89× |
(400 trajectories of 800 steps.) Replay is not a memory optimisation or a sample-efficiency trick here — not a byte of data changed, and the fit is twelve times worse without it. The rose path shows why: consecutive samples are near-duplicates, so the weights get dragged wherever the agent currently is, then dragged somewhere else when it moves on. The teal path sees a fresh mix at every step and walks more or less straight to the answer.
The second row is the part worth remembering. The damage grows with the learning rate — 3× at 0.02, 12× at 0.05 — because a bigger step lets each stretch of correlated samples pull the weights further before the next region arrives. So the obvious fix, "use a smaller learning rate", does not solve the problem; it trades it for slowness. Replay changes the order, and the order is what is broken.
Problem 2: a moving target → Target Network
In the loss, the target depends on the same parameters we're updating — chasing a target that moves every step causes oscillation or divergence. DQN keeps a separate, frozen copy for computing targets, synced to only every steps:
The two pillars of DQN
Experience replay decorrelates the data; the target network stabilizes the regression target. Remove either and DQN typically diverges. These two tricks are the whole reason deep Q-learning works.
The full loop
- Observe , pick via -greedy on .
- Execute ; store in the replay buffer.
- Sample a random minibatch; compute targets with the target net .
- Gradient-descent step on the squared TD error w.r.t. .
- Every steps, copy . Decay .
import torch, torch.nn as nn
class QNet(nn.Module):
def __init__(self, obs_dim, n_actions):
super().__init__()
self.net = nn.Sequential(
nn.Linear(obs_dim, 128), nn.ReLU(),
nn.Linear(128, 128), nn.ReLU(),
nn.Linear(128, n_actions), # one Q-value per action
)
def forward(self, x): return self.net(x)
def train_step(q, q_target, batch, optim, gamma=0.99):
s, a, r, s2, done = batch # tensors
q_sa = q(s).gather(1, a.unsqueeze(1)).squeeze(1)
with torch.no_grad(): # target net: no gradient
target = r + gamma * q_target(s2).max(1).values * (1 - done)
loss = nn.functional.mse_loss(q_sa, target)
optim.zero_grad(); loss.backward(); optim.step()
return loss.item()
A fully worked DQN training step
Setup: a 2-action gridworld (actions: left = 0, right = 1). At training step the agent samples one transition from its replay buffer:
- Online network outputs: → selected
- Target network outputs: →
- Discount:
Step 1 — TD target from the frozen target network:
Step 2 — TD error (Bellman residual):
Step 3 — Loss for this transition:
Step 4 — Gradient update (conceptually): the online network nudges downward toward , while remains frozen. After steps the weights are copied: .
gamma = 0.99
r = -1
q_online_s = [0.3, 0.7] # online Q(A, .)
q_target_s_prime = [0.5, 0.9] # target Q(B, .)
a = 1 # action taken: right
q_sa = q_online_s[a]
td_target = r + gamma * max(q_target_s_prime)
td_error = td_target - q_sa
loss = td_error ** 2
print(f"TD target: {td_target:.3f}") # -0.109
print(f"TD error: {td_error:.3f}") # -0.809
print(f"Loss: {loss:.3f}") # 0.654
Double DQN: fixing overestimation
The in the target systematically overestimates Q-values (it's a max over noisy estimates). Double DQN decouples action selection from evaluation: pick the best next action with the online net , but read its value from the target net :
A one-line change that noticeably improves stability and final performance.
The DQN family
| Variant | Adds | Benefit |
|---|---|---|
| Double DQN | Decoupled selection/evaluation | Less overestimation |
| Dueling DQN | Separate value & advantage streams | Better with many similar-value actions |
| Prioritized Replay | Sample high-error transitions more | Faster learning |
| Rainbow | Combines six improvements | State-of-the-art on Atari |
DQN is for discrete actions
The over outputs requires a finite, discrete action set. For continuous actions (robot joint torques) you can't enumerate the max — use policy-gradient / actor-critic methods (next lesson) instead.
Exercise
What problem does the experience replay buffer solve in DQN?
Need a hint?
Consecutive transitions in an episode are highly similar.
Exercise
Why does DQN use a separate, periodically-updated target network?
Need a hint?
What would happen if the target depended on the same weights being updated each step?
Exercise
What problem does Double DQN address?
Need a hint?
Think about the max operator over noisy Q-estimates.
Common mistakes
- No target network: training the network against its own moving output usually diverges.
- Replay buffer too small: recent-only data stays correlated; use a large buffer (10⁴–10⁶).
- Forgetting the terminal mask: at episode end there's no future value; multiply the bootstrap term by .
Key takeaways
- A DQN approximates with a network so it generalizes across states.
- Experience replay decorrelates data; a target network stabilizes the target.
- Double DQN reduces the max-operator's overestimation.
- DQN needs a discrete action set.
Related concepts
- Q-Learning: the tabular method DQN scales
- Policy Gradient & Actor-Critic: the approach for continuous actions
- What is a Neuron?: the function approximator inside a DQN