Vanishing & Exploding Gradients: Spectral Analysis
Each BPTT step multiplies the gradient by a Jacobian . Over a gap of steps the gradient is bounded by . This one bound explains why gradients either vanish or explode.
For the derivation of the Jacobian itself, see:
The spectral-norm bound on the gradient
The full gap- gradient is:
Apply the submultiplicativity of norms () to every factor. Let:
- (achieved at ; equals )
- — the spectral norm (largest singular value)
Each factor satisfies . Multiplying such bounds:
The exponent is the whole story.
The three regimes
| Spectral norm | Behavior | Effect on training |
|---|---|---|
| Bound exponentially | Vanishing: early steps receive near-zero gradient | |
| Bound stays flat | Neutral — but still makes it drift down | |
| Bound exponentially | Exploding: huge updates, NaN loss |
tanh' only ever makes vanishing worse
The factor satisfies , with equality only at . Once any unit is even slightly saturated (), . This multiplies the per-step gain below . It can push a borderline-stable RNN into vanishing, but it can never rescue a vanishing one.
Concrete spectral radii, numerically
Vanishing case: .
Use the diagonal matrix , whose -th power is and whose spectral norm decays like . Gradient magnitude after steps:
| Gap | Bound | % of original signal |
|---|---|---|
| 1 | ||
| 5 | ||
| 10 | ||
| 20 | ||
| 50 |
After 20 steps the signal is below of full strength — numerically zero.
Exploding case: .
Use , norm grows like :
| Gap | Bound | Growth factor |
|---|---|---|
| 1 | ||
| 5 | ||
| 10 | ||
| 20 | ||
| 50 |
Combined effect of saturated .
With a typical saturated unit giving , the per-step gain becomes . For this drops to per step:
| Gap | Effective gain |
|---|---|
| 5 | |
| 10 | |
| 20 |
The signal is gone in ten steps when units are saturated.
import math
def tanh_prime(z):
return 1 - math.tanh(z)**2
print(f"max tanh'(z) at z=0: {tanh_prime(0.0):.4f}")
print(f"tanh'(1): {tanh_prime(1.0):.4f}")
print(f"tanh'(2): {tanh_prime(2.0):.4f}")
print()
for radius in [0.5, 1.0, 1.5]:
print(f"spectral radius = {radius}")
for gap in [1, 5, 10, 20, 50]:
print(f" gap={gap:>2}: bound = {radius**gap:.3e}")
print()
# With saturation: effective per-step gain = 0.5 * 0.4 = 0.2
effective_gain = 0.5 * 0.4
print("With tanh'≈0.4 (saturated), effective gain per step:", effective_gain)
for gap in [5, 10, 20]:
print(f" gap={gap}: {effective_gain**gap:.3e}")
Spectral radius versus spectral norm
The bound above uses the spectral norm (largest singular value) for a worst-case guarantee. For the long-run growth rate along the dominant direction, use the spectral radius — the largest eigenvalue magnitude. For a product of the same matrix:
The condition for vanishing is ; for exploding, . These are exactly the same regimes — the spectral norm gives the per-step bound, the spectral radius gives the long-run exponent.
Why gradient clipping fixes explosions but not vanishing
Gradient clipping rescales an over-large gradient back to a safe norm:
This stops from corrupting the update. But it does nothing for the case — you cannot rescale a signal that has already decayed to zero. Vanishing requires an architectural fix: LSTMs and GRUs add an additive cell-state path whose per-step Jacobian is , keeping the product from shrinking.
Related concepts
- BPTT: derivation of the per-step Jacobian that is raised to the -th power here
- LSTM & GRU: gated cells with near-unit Jacobian
- Eigenvalues and Eigenvectors: spectral radius is the dominant eigenvalue magnitude