Deep Q-NetworksOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Deep Q-Networks

Tabular Q-learning dies when the state space is huge — you cannot tabulate every Atari screen. A Deep Q-Network (DQN) replaces the Q-table with a neural network Q(s,a;θ)Q(s, a; \theta) that generalizes across states. DeepMind's 2015 DQN learned to play 49 Atari games from raw pixels at human level using one architecture — the result that launched modern deep RL.

Real-world examples: Atari from pixels, game AI, robotics control, resource scheduling, the foundation for Rainbow, Double/Dueling DQN, and beyond.

From table to function approximator

Instead of storing Q(s,a)Q(s,a) in a grid, a network takes the state ss as input and outputs a Q-value for every action at once:

Q(s,⋅;θ):R∣s∣→R∣A∣Q(s, \cdot; \theta) : \mathbb{R}^{|s|} \to \mathbb{R}^{|A|}

For Atari, ss is a stack of 4 game frames, the network is a CNN, and the output is one Q-value per joystick action. Now nearby states share parameters, so the agent generalizes to screens it has never seen — impossible with a table.

Intuition

A table memorizes; a network generalizes. Two slightly different game screens produce similar features and similar Q-values automatically — the network interpolates. That's the only way to handle state spaces too large to ever visit fully.

Training as regression

We turn the Q-learning update into a supervised loss. The target is the same Bellman backup; we minimize the squared TD error:

L(θ)=E[(r+γmax⁡a′Q(s′,a′;θ−)⏟target y−Q(s,a;θ))2]\mathcal{L}(\theta) = \mathbb{E}\Big[\big(\underbrace{r + \gamma \max_{a'} Q(s', a'; \theta^-)}_{\text{target } y} - Q(s, a; \theta)\big)^2\Big]

and update θ\theta by gradient descent. But done naively this is unstable — for two deep reasons, each with a fix.

Problem 1: correlated data → Experience Replay

Consecutive transitions in an episode are highly correlated, and SGD assumes i.i.d. samples. DQN stores transitions (s,a,r,s′)(s, a, r, s') in a replay buffer and trains on random minibatches drawn from it.

Python
from collections import deque
import random

replay = deque(maxlen=100_000)
replay.append((s, a, r, s2, done))          # store every step
batch = random.sample(replay, 32)           # train on a random, decorrelated batch

Benefits: breaks temporal correlation, and reuses each experience many times (sample-efficient).

Same data, same updates, different order

true ww₁ →sequential (no replay)shuffled (replay buffer)
sequential error (this run)0.0426
shuffled error (this run)0.0044
sequential (400-run mean)0.1928
shuffled (400-run mean)0.0162
replay is better by11.89×

Raise the learning rate and the gap widens — 3.33× at 0.02, 11.89× at 0.05 — because a larger step lets each stretch of near-identical samples drag the weights further before the next region arrives. That is why “just use a smaller learning rate” is not the fix: it trades the problem for slowness. Replay changes the order, and the order is the problem.

Weight-space trajectories of the two fits. The agent walks slowly through feature space so consecutive samples are near-duplicates; the sequential run (rose) is dragged around by whichever region it is currently in, while the shuffled run (teal) sees a fresh mix at every step. The white cross is the true weight vector. Averages are over 400 independent trajectories.

"SGD assumes i.i.d. samples" is easy to nod along to and hard to feel. Here the same data is fitted twice — same learning rate, same number of updates, differing only in the order:

learning ratesequential ∥w−w∗∥\lVert w - w^*\rVertshuffledratio
0.020.25750.07743.33×
0.050.19280.016211.89×

(400 trajectories of 800 steps.) Replay is not a memory optimisation or a sample-efficiency trick here — not a byte of data changed, and the fit is twelve times worse without it. The rose path shows why: consecutive samples are near-duplicates, so the weights get dragged wherever the agent currently is, then dragged somewhere else when it moves on. The teal path sees a fresh mix at every step and walks more or less straight to the answer.

The second row is the part worth remembering. The damage grows with the learning rate — 3× at 0.02, 12× at 0.05 — because a bigger step lets each stretch of correlated samples pull the weights further before the next region arrives. So the obvious fix, "use a smaller learning rate", does not solve the problem; it trades it for slowness. Replay changes the order, and the order is what is broken.

Problem 2: a moving target → Target Network

In the loss, the target yy depends on the same parameters θ\theta we're updating — chasing a target that moves every step causes oscillation or divergence. DQN keeps a separate, frozen copy θ−\theta^- for computing targets, synced to θ\theta only every NN steps:

y=r+γmax⁡a′Q(s′,a′; θ−)θ−←θ every N stepsy = r + \gamma \max_{a'} Q(s', a';\, \theta^-) \qquad \theta^- \leftarrow \theta \text{ every } N \text{ steps}

The two pillars of DQN

Experience replay decorrelates the data; the target network stabilizes the regression target. Remove either and DQN typically diverges. These two tricks are the whole reason deep Q-learning works.

The full loop

  1. Observe ss, pick aa via ε\varepsilon-greedy on Q(s,⋅;θ)Q(s,\cdot;\theta).
  2. Execute aa; store (s,a,r,s′,done)(s, a, r, s', \text{done}) in the replay buffer.
  3. Sample a random minibatch; compute targets with the target net θ−\theta^-.
  4. Gradient-descent step on the squared TD error w.r.t. θ\theta.
  5. Every NN steps, copy θ→θ−\theta \to \theta^-. Decay ε\varepsilon.
Python
import torch, torch.nn as nn

class QNet(nn.Module):
    def __init__(self, obs_dim, n_actions):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(obs_dim, 128), nn.ReLU(),
            nn.Linear(128, 128), nn.ReLU(),
            nn.Linear(128, n_actions),       # one Q-value per action
        )
    def forward(self, x): return self.net(x)

def train_step(q, q_target, batch, optim, gamma=0.99):
    s, a, r, s2, done = batch                 # tensors
    q_sa = q(s).gather(1, a.unsqueeze(1)).squeeze(1)
    with torch.no_grad():                      # target net: no gradient
        target = r + gamma * q_target(s2).max(1).values * (1 - done)
    loss = nn.functional.mse_loss(q_sa, target)
    optim.zero_grad(); loss.backward(); optim.step()
    return loss.item()

A fully worked DQN training step

Setup: a 2-action gridworld (actions: left = 0, right = 1). At training step tt the agent samples one transition from its replay buffer:

  • (s,a,r,s′)=(state A, right, −1, state B)(s, a, r, s') = (\text{state A},\ \text{right},\ -1,\ \text{state B})
  • Online network outputs: Qθ(A)=[0.3, 0.7]Q_\theta(\text{A}) = [0.3,\ 0.7] → selected Qθ(A,right)=0.7Q_\theta(\text{A}, \text{right}) = 0.7
  • Target network outputs: Qθ−(B)=[0.5, 0.9]Q_{\theta^-}(\text{B}) = [0.5,\ 0.9] → max⁡=0.9\max = 0.9
  • Discount: γ=0.99\gamma = 0.99

Step 1 — TD target from the frozen target network:

y=r+γmax⁡a′Qθ−(s′,a′)=−1+0.99×0.9=−1+0.891=−0.109y = r + \gamma \max_{a'} Q_{\theta^-}(s', a') = -1 + 0.99 \times 0.9 = -1 + 0.891 = -0.109

Step 2 — TD error (Bellman residual):

δ=y−Qθ(s,a)=−0.109−0.7=−0.809\delta = y - Q_\theta(s, a) = -0.109 - 0.7 = -0.809

Step 3 — Loss for this transition:

L=δ2=(−0.809)2=0.654\mathcal{L} = \delta^2 = (-0.809)^2 = 0.654

Step 4 — Gradient update (conceptually): the online network nudges Qθ(A,right)Q_\theta(\text{A}, \text{right}) downward toward −0.109-0.109, while Qθ−Q_{\theta^-} remains frozen. After CC steps the weights are copied: θ→θ−\theta \to \theta^-.

Python
gamma = 0.99
r = -1
q_online_s = [0.3, 0.7]        # online Q(A, .)
q_target_s_prime = [0.5, 0.9]  # target Q(B, .)
a = 1                           # action taken: right

q_sa = q_online_s[a]
td_target = r + gamma * max(q_target_s_prime)
td_error  = td_target - q_sa
loss      = td_error ** 2

print(f"TD target: {td_target:.3f}")   # -0.109
print(f"TD error:  {td_error:.3f}")    # -0.809
print(f"Loss:      {loss:.3f}")        # 0.654

Double DQN: fixing overestimation

The max⁡\max in the target systematically overestimates Q-values (it's a max over noisy estimates). Double DQN decouples action selection from evaluation: pick the best next action with the online net θ\theta, but read its value from the target net θ−\theta^-:

y=r+γ Q(s′, arg⁡max⁡a′Q(s′,a′;θ); θ−)y = r + \gamma\, Q\big(s',\ \arg\max_{a'} Q(s', a'; \theta);\ \theta^-\big)

A one-line change that noticeably improves stability and final performance.

The DQN family

VariantAddsBenefit
Double DQNDecoupled selection/evaluationLess overestimation
Dueling DQNSeparate value & advantage streamsBetter with many similar-value actions
Prioritized ReplaySample high-error transitions moreFaster learning
RainbowCombines six improvementsState-of-the-art on Atari

DQN is for discrete actions

The max⁡a\max_a over outputs requires a finite, discrete action set. For continuous actions (robot joint torques) you can't enumerate the max — use policy-gradient / actor-critic methods (next lesson) instead.

Exercise

What problem does the experience replay buffer solve in DQN?

Need a hint?

Consecutive transitions in an episode are highly similar.

Exercise

Why does DQN use a separate, periodically-updated target network?

Need a hint?

What would happen if the target depended on the same weights being updated each step?

Exercise

What problem does Double DQN address?

Need a hint?

Think about the max operator over noisy Q-estimates.

Common mistakes

  1. No target network: training the network against its own moving output usually diverges.
  2. Replay buffer too small: recent-only data stays correlated; use a large buffer (10⁴–10⁶).
  3. Forgetting the terminal mask: at episode end there's no future value; multiply the bootstrap term by (1−done)(1-\text{done}).

Key takeaways

  • A DQN approximates QQ with a network so it generalizes across states.
  • Experience replay decorrelates data; a target network stabilizes the target.
  • Double DQN reduces the max-operator's overestimation.
  • DQN needs a discrete action set.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelated
Deep Q-NetworksNeural NetworksProbabilistic Models & EMQ-LearningPolicy Gradient & Actor-C…What is a Neuron?