Vanishing & Exploding Gradients: Spectral AnalysisOpen in Colab

Vanishing & Exploding Gradients: Spectral Analysis

Each BPTT step multiplies the gradient by a Jacobian Jt=diag(tanh⁡′(at)) WhhJ_t = \mathrm{diag}(\tanh'(\mathbf{a}_t))\,W_{hh}. Over a gap of T−kT-k steps the gradient is bounded by ∥J∥T−k\|J\|^{T-k}. This one bound explains why gradients either vanish or explode.

For the derivation of the Jacobian itself, see:

Deep dive
Backpropagation Through Time (BPTT)
Includes runnable notebook

The spectral-norm bound on the gradient

The full gap-(T−k)(T-k) gradient is:

∂LT∂hk=∂LT∂hT∏t=k+1Tdiag ⁣(tanh⁡′(at)) Whh\frac{\partial \mathcal{L}_T}{\partial \mathbf{h}_k} = \frac{\partial \mathcal{L}_T}{\partial \mathbf{h}_T} \prod_{t=k+1}^{T} \mathrm{diag}\!\big(\tanh'(\mathbf{a}_t)\big)\,W_{hh}

Apply the submultiplicativity of norms (∥AB∥≤∥A∥ ∥B∥\|AB\| \le \|A\|\,\|B\|) to every factor. Let:

  • γ=max⁡ztanh⁡′(z)=1\gamma = \max_z \tanh'(z) = 1 (achieved at z=0z = 0; equals 1−tanh⁡2(z)1 - \tanh^2(z))
  • λ=∥Whh∥\lambda = \|W_{hh}\| — the spectral norm (largest singular value)

Each factor satisfies ∥diag(tanh⁡′) Whh∥≤γ⋅λ=λ\|\mathrm{diag}(\tanh')\,W_{hh}\| \le \gamma \cdot \lambda = \lambda. Multiplying T−kT-k such bounds:

∥∂LT∂hk∥  ≤  ∥∂LT∂hT∥⋅λT−k\left\|\frac{\partial \mathcal{L}_T}{\partial \mathbf{h}_k}\right\| \;\le\; \left\|\frac{\partial \mathcal{L}_T}{\partial \mathbf{h}_T}\right\| \cdot \lambda^{T-k}

The exponent T−kT-k is the whole story.

The three regimes

Spectral norm λ=∥Whh∥\lambda = \|W_{hh}\|BehaviorEffect on training
λ<1\lambda < 1Bound →0\to 0 exponentiallyVanishing: early steps receive near-zero gradient
λ=1\lambda = 1Bound stays flatNeutral — but tanh⁡′≤1\tanh' \le 1 still makes it drift down
λ>1\lambda > 1Bound →∞\to \infty exponentiallyExploding: huge updates, NaN loss

tanh' only ever makes vanishing worse

The tanh⁡′\tanh' factor satisfies 0<tanh⁡′(z)≤10 < \tanh'(z) \le 1, with equality only at z=0z = 0. Once any unit is even slightly saturated (∣z∣>1|z| > 1), tanh⁡′(z)<0.42\tanh'(z) < 0.42. This multiplies the per-step gain below λ\lambda. It can push a borderline-stable RNN into vanishing, but it can never rescue a vanishing one.

Concrete spectral radii, numerically

Vanishing case: λ=0.5\lambda = 0.5.

Use the diagonal matrix Whh=diag(0.5,0.3)W_{hh} = \mathrm{diag}(0.5, 0.3), whose nn-th power is diag(0.5n,0.3n)\mathrm{diag}(0.5^n, 0.3^n) and whose spectral norm decays like 0.5n0.5^n. Gradient magnitude after n=T−kn = T-k steps:

Gap nnBound 0.5n0.5^n% of original signal
10.5000.50050%50\%
50.0310.0313.1%3.1\%
109.8×10−49.8 \times 10^{-4}0.098%0.098\%
209.5×10−79.5 \times 10^{-7}0.000095%0.000095\%
508.9×10−168.9 \times 10^{-16}≈0\approx 0

After 20 steps the signal is below 0.0001%0.0001\% of full strength — numerically zero.

Exploding case: λ=1.5\lambda = 1.5.

Use Whh=diag(1.5,1.2)W_{hh} = \mathrm{diag}(1.5, 1.2), norm grows like 1.5n1.5^n:

Gap nnBound 1.5n1.5^nGrowth factor
11.51.51.5×1.5\times
57.67.67.6×7.6\times
1057.757.757.7×57.7\times
203.3×1033.3 \times 10^{3}3300×3300\times
506.4×1086.4 \times 10^{8}640,000,000×640{,}000{,}000\times

Combined effect of saturated tanh⁡′\tanh'.

With a typical saturated unit giving tanh⁡′≈0.4\tanh' \approx 0.4, the per-step gain becomes λ×0.4\lambda \times 0.4. For λ=0.5\lambda = 0.5 this drops to 0.20.2 per step:

Gap nnEffective gain (0.5×0.4)n=0.2n(0.5 \times 0.4)^n = 0.2^n
53.2×10−43.2 \times 10^{-4}
10≈10−7\approx 10^{-7}
20≈10−14\approx 10^{-14}

The signal is gone in ten steps when units are saturated.

Python
import math

def tanh_prime(z):
    return 1 - math.tanh(z)**2

print(f"max tanh'(z) at z=0:   {tanh_prime(0.0):.4f}")
print(f"tanh'(1):               {tanh_prime(1.0):.4f}")
print(f"tanh'(2):               {tanh_prime(2.0):.4f}")
print()

for radius in [0.5, 1.0, 1.5]:
    print(f"spectral radius = {radius}")
    for gap in [1, 5, 10, 20, 50]:
        print(f"  gap={gap:>2}:  bound = {radius**gap:.3e}")
    print()

# With saturation: effective per-step gain = 0.5 * 0.4 = 0.2
effective_gain = 0.5 * 0.4
print("With tanh'≈0.4 (saturated), effective gain per step:", effective_gain)
for gap in [5, 10, 20]:
    print(f"  gap={gap}: {effective_gain**gap:.3e}")

Spectral radius versus spectral norm

The bound above uses the spectral norm ∥Whh∥\|W_{hh}\| (largest singular value) for a worst-case guarantee. For the long-run growth rate along the dominant direction, use the spectral radius ρ(Whh)\rho(W_{hh}) — the largest eigenvalue magnitude. For a product of the same matrix:

∥Whhn∥∼ρ(Whh)nas n→∞\|W_{hh}^n\| \sim \rho(W_{hh})^n \quad \text{as } n \to \infty

The condition for vanishing is ρ(Whh)<1\rho(W_{hh}) < 1; for exploding, ρ(Whh)>1\rho(W_{hh}) > 1. These are exactly the same regimes — the spectral norm gives the per-step bound, the spectral radius gives the long-run exponent.

Why gradient clipping fixes explosions but not vanishing

Gradient clipping rescales an over-large gradient back to a safe norm:

if ∥g∥>c:g←c∥g∥ g\text{if } \|\mathbf{g}\| > c:\quad \mathbf{g} \leftarrow \frac{c}{\|\mathbf{g}\|}\,\mathbf{g}

This stops λT−k→∞\lambda^{T-k} \to \infty from corrupting the update. But it does nothing for the λT−k→0\lambda^{T-k} \to 0 case — you cannot rescale a signal that has already decayed to zero. Vanishing requires an architectural fix: LSTMs and GRUs add an additive cell-state path whose per-step Jacobian is ≈1\approx 1, keeping the product from shrinking.

Related concepts

  • BPTT: derivation of the per-step Jacobian that is raised to the (T−k)(T-k)-th power here
  • LSTM & GRU: gated cells with near-unit Jacobian
  • Eigenvalues and Eigenvectors: spectral radius is the dominant eigenvalue magnitude

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

RelatedReferenced by
Vanishing & Exploding Gradients: …Activation Functions: Fro…Automatic Differentiation…Backpropagation Through T…Computing Eigenvalues & E…BPTT & the Vanishing Grad…LSTM & GRUEigenvalues and Eigenvect…