Backpropagation Through Time & the Vanishing Gradient
An RNN learns the same way any network does — gradient descent on a loss. But because the network is a loop unrolled over time, the gradient has to flow backward through every time step. This is Backpropagation Through Time (BPTT), and it exposes the single biggest weakness of vanilla RNNs.
Unrolling for training
To train, we unroll the RNN over the whole sequence, treating it as a very deep feed-forward network (one "layer" per time step) that happens to share weights. The total loss sums each step's loss:
Because is reused at every step, its gradient is the sum of contributions from all steps:
Intuition
BPTT is ordinary backprop on the unrolled graph. The twist: one weight matrix appears in every layer, so its update gathers a vote from every time step. Early steps get a say in the final prediction — if their gradient survives the trip back.
The chain through time
The gradient requires threading the chain rule through every hidden state between and , producing a product of Jacobians. One step's Jacobian is — where is the element-wise derivative of tanh (a vector of values in ), and builds a diagonal matrix from that vector so the multiplication scales each row of independently. Each step back multiplies by one copy of and one diagonal of values. For the full derivation and a 3-step scalar worked trace (all distances, cumulative products, and numerical verification), see:
Vanishing & exploding gradients through time
Each timestep multiplies the gradient by the same factor (weight × activation slope). Reaching an early step means multiplying it ~20 times — factor < 1 vanishes toward zero, factor > 1 explodes. This is why long-range learning is hard, and why LSTMs/GRUs exist.
Why gradients vanish (or explode)
Applying the submultiplicativity of norms to the product of Jacobians bounds the gap- gradient magnitude by , where is the spectral norm. When gradients vanish exponentially; when they explode. The factor (always ) only ever makes vanishing worse. For the full spectral-norm derivation, the three-regime table, and numeric decay/growth tables at for gaps up to 50, see:
Exploding gradients: gradient clipping
Exploding gradients have a cheap, standard fix — clip the gradient norm before each update:
import torch.nn as nn
# after loss.backward(), before optimizer.step():
nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
Clipping fixes explosions, not vanishing
Gradient clipping rescales an over-large gradient back to a safe norm — it stops explosions and NaNs. It does nothing for vanishing gradients: you cannot rescale a signal that has already decayed to zero. That problem needs an architectural fix.
Truncated BPTT
For very long sequences, unrolling the entire history is too expensive in memory. Truncated BPTT backpropagates only through the last steps, then detaches the hidden state so the graph does not grow forever:
hidden = hidden.detach() # keep the value, drop the gradient history
The real fix is architectural
You cannot tune your way out of vanishing gradients: the bound (where is the tanh derivative factor) is exponential in the gap no matter how you initialize, because every path back multiplies by and . The fix is to change the architecture so there is a path whose per-step Jacobian is instead of . That is exactly what LSTMs and GRUs do: their cell state is updated additively (), so and the product of Jacobians stays near 1 over long gaps — a gradient highway. The next lesson derives those gates.
| Problem | Symptom | Fix |
|---|---|---|
| Exploding gradient | Loss → NaN, wild swings | Gradient clipping |
| Vanishing gradient | Loss plateaus, no long-range learning | LSTM / GRU (gated cells) |
Exercise
In BPTT, the gradient across a gap of T−k steps is a product of T−k Jacobians. If each factor has magnitude ~0.7, what happens over 20 steps?
Need a hint?
Compute 0.7 raised to the 20th power.
Exercise
What is the root cause of the vanishing gradient problem in vanilla RNNs?
Need a hint?
Think about what happens when you multiply many similar matrices together.
Exercise
Gradient clipping addresses which problem in training RNNs?
Need a hint?
It rescales gradients that are too large.
Common mistakes
- Clipping to "fix" vanishing: clipping only addresses explosions.
- Forgetting to
detach()the hidden state in truncated BPTT — backprop graph grows without bound and OOMs. - Blaming the learning rate when the model can't learn long-range patterns — it's the architecture, not the step size.
Key takeaways
- BPTT is backprop on the unrolled network; one weight matrix appears at every step.
- One step's Jacobian is ; the gap- gradient is a product of of these.
- That product is bounded by , where , so gradients vanish (gain below 1) or explode (gain above 1) exponentially in the gap.
- only ever makes vanishing worse, never lifts it back.
- Gradient clipping fixes explosions but not vanishing; vanishing needs an architectural fix — gated cells whose Jacobian is .
Related concepts
- Recurrent Neural Networks: the model being trained
- LSTM & GRU: the gated cells that solve vanishing gradients
- Gradient Descent: the base optimization being extended through time