BPTT & the Vanishing GradientOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Backpropagation Through Time & the Vanishing Gradient

An RNN learns the same way any network does — gradient descent on a loss. But because the network is a loop unrolled over time, the gradient has to flow backward through every time step. This is Backpropagation Through Time (BPTT), and it exposes the single biggest weakness of vanilla RNNs.

Unrolling for training

To train, we unroll the RNN over the whole sequence, treating it as a very deep feed-forward network (one "layer" per time step) that happens to share weights. The total loss sums each step's loss:

L=∑t=1TLt(yt,y^t)\mathcal{L} = \sum_{t=1}^{T} \mathcal{L}_t(\mathbf{y}_t, \hat{\mathbf{y}}_t)

Because WhhW_{hh} is reused at every step, its gradient is the sum of contributions from all steps:

∂L∂Whh=∑t=1T∂Lt∂Whh\frac{\partial \mathcal{L}}{\partial W_{hh}} = \sum_{t=1}^{T} \frac{\partial \mathcal{L}_t}{\partial W_{hh}}

Intuition

BPTT is ordinary backprop on the unrolled graph. The twist: one weight matrix appears in every layer, so its update gathers a vote from every time step. Early steps get a say in the final prediction — if their gradient survives the trip back.

The chain through time

The gradient ∂LT/∂hk\partial \mathcal{L}_T / \partial \mathbf{h}_k requires threading the chain rule through every hidden state between kk and TT, producing a product of T−kT-k Jacobians. One step's Jacobian is ∂ht/∂ht−1=diag(tanh⁡′(at)) Whh\partial \mathbf{h}_t/\partial \mathbf{h}_{t-1} = \mathrm{diag}(\tanh'(\mathbf{a}_t))\,W_{hh} — where tanh⁡′(at)\tanh'(\mathbf{a}_t) is the element-wise derivative of tanh (a vector of values in [0,1][0,1]), and diag(⋅)\mathrm{diag}(\cdot) builds a diagonal matrix from that vector so the multiplication scales each row of WhhW_{hh} independently. Each step back multiplies by one copy of WhhW_{hh} and one diagonal of tanh⁡′\tanh' values. For the full derivation and a 3-step scalar worked trace (all distances, cumulative products, and numerical verification), see:

Deep dive
Backpropagation Through Time (BPTT)
Includes runnable notebook

Vanishing & exploding gradients through time

timestep (← earlier · output →)grad magnitude (log)1e01e-31e-61e-9
regimevanishing
earliest grad1.1e-3
ratio (latest/earliest)877.28

Each timestep multiplies the gradient by the same factor (weight × activation slope). Reaching an early step means multiplying it ~20 times — factor < 1 vanishes toward zero, factor > 1 explodes. This is why long-range learning is hard, and why LSTMs/GRUs exist.

Why gradients vanish (or explode)

Applying the submultiplicativity of norms to the product of Jacobians bounds the gap-(T−k)(T-k) gradient magnitude by λT−k\lambda^{T-k}, where λ=∥Whh∥\lambda = \|W_{hh}\| is the spectral norm. When λ<1\lambda < 1 gradients vanish exponentially; when λ>1\lambda > 1 they explode. The tanh⁡′\tanh' factor (always ≤1\le 1) only ever makes vanishing worse. For the full spectral-norm derivation, the three-regime table, and numeric decay/growth tables at λ∈{0.5,1.0,1.5}\lambda \in \{0.5, 1.0, 1.5\} for gaps up to 50, see:

Deep dive
Vanishing & Exploding Gradients: Spectral Analysis
Includes runnable notebook

Exploding gradients: gradient clipping

Exploding gradients have a cheap, standard fix — clip the gradient norm before each update:

if ∥g∥>c:g←c∥g∥ g\text{if } \|\mathbf{g}\| > c:\quad \mathbf{g} \leftarrow \frac{c}{\|\mathbf{g}\|}\,\mathbf{g}
Python
import torch.nn as nn

# after loss.backward(), before optimizer.step():
nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)

Clipping fixes explosions, not vanishing

Gradient clipping rescales an over-large gradient back to a safe norm — it stops explosions and NaNs. It does nothing for vanishing gradients: you cannot rescale a signal that has already decayed to zero. That problem needs an architectural fix.

Truncated BPTT

For very long sequences, unrolling the entire history is too expensive in memory. Truncated BPTT backpropagates only through the last kk steps, then detaches the hidden state so the graph does not grow forever:

Python
hidden = hidden.detach()   # keep the value, drop the gradient history

The real fix is architectural

You cannot tune your way out of vanishing gradients: the bound (∥Whh∥⋅γ)T−k(\|W_{hh}\|\cdot\gamma)^{T-k} (where γ=max⁡t∥diag(tanh⁡′(at))∥≤1\gamma = \max_t \|\mathrm{diag}(\tanh'(\mathbf{a}_t))\| \le 1 is the tanh derivative factor) is exponential in the gap no matter how you initialize, because every path back multiplies by WhhW_{hh} and tanh⁡′\tanh'. The fix is to change the architecture so there is a path whose per-step Jacobian is ≈1\approx 1 instead of ∥Whh∥⋅γ\|W_{hh}\|\cdot\gamma. That is exactly what LSTMs and GRUs do: their cell state is updated additively (ct≈ct−1+(stuff)c_t \approx c_{t-1} + \text{(stuff)}), so ∂ct/∂ct−1≈1\partial c_t/\partial c_{t-1} \approx 1 and the product of Jacobians stays near 1 over long gaps — a gradient highway. The next lesson derives those gates.

ProblemSymptomFix
Exploding gradientLoss → NaN, wild swingsGradient clipping
Vanishing gradientLoss plateaus, no long-range learningLSTM / GRU (gated cells)

Exercise

In BPTT, the gradient across a gap of T−k steps is a product of T−k Jacobians. If each factor has magnitude ~0.7, what happens over 20 steps?

Need a hint?

Compute 0.7 raised to the 20th power.

Exercise

What is the root cause of the vanishing gradient problem in vanilla RNNs?

Need a hint?

Think about what happens when you multiply many similar matrices together.

Exercise

Gradient clipping addresses which problem in training RNNs?

Need a hint?

It rescales gradients that are too large.

Common mistakes

  1. Clipping to "fix" vanishing: clipping only addresses explosions.
  2. Forgetting to detach() the hidden state in truncated BPTT — backprop graph grows without bound and OOMs.
  3. Blaming the learning rate when the model can't learn long-range patterns — it's the architecture, not the step size.

Key takeaways

  • BPTT is backprop on the unrolled network; one weight matrix appears at every step.
  • One step's Jacobian is ∂ht/∂ht−1=diag(tanh⁡′(at)) Whh\partial\mathbf{h}_t/\partial\mathbf{h}_{t-1} = \mathrm{diag}(\tanh'(\mathbf{a}_t))\,W_{hh}; the gap-(T−k)(T-k) gradient is a product of T−kT-k of these.
  • That product is bounded by (∥Whh∥⋅γ)T−k(\|W_{hh}\|\cdot\gamma)^{T-k}, where γ=max⁡t∥diag(tanh⁡′(at))∥≤1\gamma = \max_t \|\mathrm{diag}(\tanh'(\mathbf{a}_t))\| \le 1, so gradients vanish (gain below 1) or explode (gain above 1) exponentially in the gap.
  • tanh⁡′≤1\tanh' \le 1 only ever makes vanishing worse, never lifts it back.
  • Gradient clipping fixes explosions but not vanishing; vanishing needs an architectural fix — gated cells whose Jacobian is ≈1\approx 1.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelatedGo deeper
BPTT & the Vanishing GradientNeural NetworksCalculus for MLGradient DescentWeight Initialization & T…Recurrent Neural NetworksLSTM & GRUState Space Models (S4 & …Backpropagation Through T…Vanishing & Exploding Gra…