Chain Rule and BackpropagationOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Chain Rule and Backpropagation

Backpropagation is not magic — it's the chain rule applied to computational graphs. It's what makes the loop's optimization stage tractable for deep nets: without an efficient way to get the gradient of the loss w.r.t. millions of parameters, gradient descent would be a non-starter. Once you understand the chain rule, every step of backprop becomes a mechanical application of one idea.

Real-world examples: Backpropagation (the algorithm that trains every deep neural network) is the chain rule applied recursively through a computational graph.

The chain rule

When a function is composed of other functions, the chain rule tells you how to differentiate the composition:

ddxf(g(x))=f′(g(x))⋅g′(x)=dfdg⋅dgdx\frac{d}{dx} f(g(x)) = f'(g(x)) \cdot g'(x) = \frac{df}{dg} \cdot \frac{dg}{dx}

Example: h(x)=σ(3x2)h(x) = \sigma(3x^2) where σ\sigma is sigmoid.

Let g=3x2g = 3x^2 and f(g)=σ(g)f(g) = \sigma(g):

h′(x)=σ′(g)⋅6x=σ(3x2)(1−σ(3x2))⋅6xh'(x) = \sigma'(g) \cdot 6x = \sigma(3x^2)(1 - \sigma(3x^2)) \cdot 6x

The trick that powers backprop is to read this right to left. We never write the whole derivative as one giant formula. Instead each elementary operation contributes one local factor, and we multiply those factors together:

dhdx⏟what we want=dhdg⏟σ′(g)⋅dgdx⏟6x\underbrace{\frac{dh}{dx}}_{\text{what we want}} = \underbrace{\frac{dh}{dg}}_{\sigma'(g)} \cdot \underbrace{\frac{dg}{dx}}_{6x}

If a third operation were stacked on top, say p=h2p = h^2, we would simply prepend one more factor: dpdx=dpdh⋅dhdg⋅dgdx\frac{dp}{dx} = \frac{dp}{dh}\cdot\frac{dh}{dg}\cdot\frac{dg}{dx}. Backprop is nothing more than this — a chain of single-operation derivatives multiplied together.

Multivariable chain rule

For z=f(x,y)z = f(x, y) where x=x(t)x = x(t) and y=y(t)y = y(t):

dzdt=∂f∂xdxdt+∂f∂ydydt\frac{dz}{dt} = \frac{\partial f}{\partial x}\frac{dx}{dt} + \frac{\partial f}{\partial y}\frac{dy}{dt}

When zz depends on x∈Rn\mathbf{x} \in \mathbb{R}^n and x\mathbf{x} depends on w\mathbf{w} via x=g(w)\mathbf{x} = g(\mathbf{w}), the chain rule becomes a matrix product of Jacobians:

∂z∂w=∂z∂x⋅∂x∂w\frac{\partial z}{\partial \mathbf{w}} = \frac{\partial z}{\partial \mathbf{x}} \cdot \frac{\partial \mathbf{x}}{\partial \mathbf{w}}

Computational graphs

A computational graph represents a function as a directed acyclic graph (DAG): nodes are operations, edges carry values forward (forward pass) and gradients backward (backward pass).

Backprop = the chain rule on a graph

squaresinx1.50u = x²2.25y = sin u0.778node values (forward pass)
forward y0.778
gradient dy/dx-1.885

Forward (teal) computes values left → right. Backward (orange) multiplies local derivatives right → left: dy/dx = dy/du · du/dx = cos(u) · 2x. Each node just multiplies the incoming gradient by its own local derivative.

Example graph: L=(w⊤x−y)2L = (\mathbf{w}^\top \mathbf{x} - y)^2

Code
x, w → [dot product] → z
z, y → [subtract]   → r = z - y
r    → [square]      → L = r²

Forward pass

Compute each node's output in topological order:

z=w⊤x,r=z−y,L=r2z = \mathbf{w}^\top \mathbf{x}, \quad r = z - y, \quad L = r^2

Backward pass

Starting from ∂L∂L=1\frac{\partial L}{\partial L} = 1, apply the chain rule in reverse:

∂L∂r=2r=2(z−y)\frac{\partial L}{\partial r} = 2r = 2(z - y) ∂L∂z=∂L∂r⋅∂r∂z=2(z−y)⋅1=2(z−y)\frac{\partial L}{\partial z} = \frac{\partial L}{\partial r} \cdot \frac{\partial r}{\partial z} = 2(z-y) \cdot 1 = 2(z-y) ∂L∂w=∂L∂z⋅∂z∂w=2(z−y)⋅x\frac{\partial L}{\partial \mathbf{w}} = \frac{\partial L}{\partial z} \cdot \frac{\partial z}{\partial \mathbf{w}} = 2(z-y) \cdot \mathbf{x}

Local gradients × upstream gradient

Each node stores its local gradient (derivative of output w.r.t. its inputs). During backprop, it multiplies the incoming upstream gradient by the local gradient and passes it back. The chain rule is always: local × upstream.

A fully worked numeric pass

Let's run the squared-error graph with real numbers so every multiplication is concrete. Take a single weight ww and feature xx, so z=wxz = wx (a one-element dot product), with

w=2,x=3,y=5.w = 2, \quad x = 3, \quad y = 5.

Forward pass — compute and cache each intermediate value:

z=wx=2⋅3=6,r=z−y=6−5=1,L=r2=1.z = wx = 2 \cdot 3 = 6, \qquad r = z - y = 6 - 5 = 1, \qquad L = r^2 = 1.

Backward pass — start at the output with ∂L∂L=1\frac{\partial L}{\partial L} = 1 and walk back, multiplying upstream × local at each node:

∂L∂r=1⏟upstream⋅2r⏟local=2⋅1=2\frac{\partial L}{\partial r} = \underbrace{1}_{\text{upstream}} \cdot \underbrace{2r}_{\text{local}} = 2 \cdot 1 = 2 ∂L∂z=2⏟∂L/∂r⋅1⏟∂r/∂z=2\frac{\partial L}{\partial z} = \underbrace{2}_{\partial L/\partial r} \cdot \underbrace{1}_{\partial r/\partial z} = 2 ∂L∂w=2⏟∂L/∂z⋅x⏟∂z/∂w=2⋅3=6\frac{\partial L}{\partial w} = \underbrace{2}_{\partial L/\partial z} \cdot \underbrace{x}_{\partial z/\partial w} = 2 \cdot 3 = 6

So ∂L∂w=6\frac{\partial L}{\partial w} = 6. We can sanity-check against the closed form L=(wx−y)2⇒∂L∂w=2(wx−y)x=2(1)(3)=6L = (wx-y)^2 \Rightarrow \frac{\partial L}{\partial w} = 2(wx-y)x = 2(1)(3) = 6. ✅ The notebook reproduces this and confirms it against a finite-difference estimate.

Worked example: one hidden layer

Now scale the exact same procedure up to a tiny MLP. Network:

y^=σ(w2⊤ ReLU(W1x+b1)+b2),L=−yln⁡y^−(1−y)ln⁡(1−y^).\hat{y} = \sigma\big(\mathbf{w}_2^\top \,\text{ReLU}(\mathbf{W}_1 \mathbf{x} + \mathbf{b}_1) + b_2\big), \qquad L = -y \ln\hat{y} - (1-y)\ln(1-\hat{y}).

Name every intermediate node so we can attach a gradient to each one:

z1=W1x+b1,a1=ReLU(z1),z2=w2⊤a1+b2,y^=σ(z2).\mathbf{z}_1 = \mathbf{W}_1\mathbf{x} + \mathbf{b}_1, \quad \mathbf{a}_1 = \text{ReLU}(\mathbf{z}_1), \quad z_2 = \mathbf{w}_2^\top\mathbf{a}_1 + b_2, \quad \hat{y} = \sigma(z_2).

The backward pass is one chain-rule multiplication per node, output → input. The first step uses a classic simplification: for binary cross-entropy composed with sigmoid, the two local gradients collapse to a single clean expression,

∂L∂z2=∂L∂y^⏟y^−yy^(1−y^)⋅∂y^∂z2⏟y^(1−y^)=y^−y.\frac{\partial L}{\partial z_2} = \underbrace{\frac{\partial L}{\partial \hat{y}}}_{\frac{\hat{y}-y}{\hat{y}(1-\hat{y})}} \cdot \underbrace{\frac{\partial \hat{y}}{\partial z_2}}_{\hat{y}(1-\hat{y})} = \hat{y} - y.

From there each subsequent line is upstream × local:

∂L∂w2=∂L∂z2 a1,∂L∂b2=∂L∂z2,∂L∂a1=∂L∂z2 w2\frac{\partial L}{\partial \mathbf{w}_2} = \frac{\partial L}{\partial z_2}\,\mathbf{a}_1, \qquad \frac{\partial L}{\partial b_2} = \frac{\partial L}{\partial z_2}, \qquad \frac{\partial L}{\partial \mathbf{a}_1} = \frac{\partial L}{\partial z_2}\,\mathbf{w}_2 ∂L∂z1=∂L∂a1⊙ReLU′(z1),∂L∂W1=∂L∂z1 x⊤,∂L∂b1=∂L∂z1\frac{\partial L}{\partial \mathbf{z}_1} = \frac{\partial L}{\partial \mathbf{a}_1}\odot \text{ReLU}'(\mathbf{z}_1), \qquad \frac{\partial L}{\partial \mathbf{W}_1} = \frac{\partial L}{\partial \mathbf{z}_1}\,\mathbf{x}^\top, \qquad \frac{\partial L}{\partial \mathbf{b}_1} = \frac{\partial L}{\partial \mathbf{z}_1}

where ⊙\odot is element-wise product and ReLU′(z1)\text{ReLU}'(\mathbf{z}_1) is 11 where z1>0\mathbf{z}_1 > 0 and 00 elsewhere. Notice how the cached forward values (a1\mathbf{a}_1, w2\mathbf{w}_2, z1\mathbf{z}_1, x\mathbf{x}) are exactly what each local gradient needs — that reuse is why one backward pass suffices.

Python
import numpy as np

def sigmoid(z): return 1 / (1 + np.exp(-z))
def relu(z):    return np.maximum(0, z)

# Forward pass
np.random.seed(0)                 # reproducible
x  = np.array([1.0, 2.0, 3.0])
W1 = np.random.randn(4, 3) * 0.1
b1 = np.zeros(4)
w2 = np.random.randn(4) * 0.1
b2 = 0.0
y  = 1.0

z1    = W1 @ x + b1      # pre-activation layer 1
a1    = relu(z1)          # activation layer 1
z2    = w2 @ a1 + b2     # pre-activation output
y_hat = sigmoid(z2)       # predicted probability
L     = -(y * np.log(y_hat) + (1-y) * np.log(1-y_hat))

# Backward pass — one line per node, upstream × local
dL_dz2 = y_hat - y                          # BCE + sigmoid collapse
dL_dw2 = dL_dz2 * a1                         # × a1
dL_db2 = dL_dz2                              # × 1
dL_da1 = dL_dz2 * w2                         # × w2
dL_dz1 = dL_da1 * (z1 > 0)                  # × ReLU'(z1)
dL_dW1 = np.outer(dL_dz1, x)                # × xᵀ
dL_db1 = dL_dz1                             # × 1

Watch both passes run

The graph above shows the shape; this steps the numbers through it — first the scalar graph worked by hand, then the same procedure on this network, with every gradient checked against a finite difference at the end:

Backpropagation through a computational graph, checked against finite differences

1/10

Start with the scalar graph the lesson works by hand: L = (wx − y)² with w = 2, x = 3, y = 5. The forward pass walks the graph in topological order and — this is the part that matters — **caches every intermediate**: z = 6, r = 1, L = 1. Nothing is thrown away, because each of those values is about to be needed again on the way back.

python
1def scalar_graph(w, x, y):
2 # forward: cache every node
3 z = w * x
4 r = z - y
5 L = r * r
6 # backward: upstream x local
7 dr = 2 * r
8 dz = dr * 1
9 dw = dz * x
10 return L, dw
11
12def forward(x, W1, b1, w2, b2, y):
13 z1 = W1 @ x + b1
14 a1 = relu(z1)
15 z2 = w2 @ a1 + b2
16 yh = sigmoid(z2)
17 L = -(y*log(yh) + (1-y)*log(1-yh))
18 return L, (z1, a1, z2, yh)
19
20def backward(x, w2, y, cache):
21 z1, a1, z2, yh = cache
22 # BCE + sigmoid collapse to yh - y
23 dz2 = yh - y
24 dw2 = dz2 * a1
25 db2 = dz2
26 da1 = dz2 * w2
27 dz1 = da1 * (z1 > 0)
28 dW1 = outer(dz1, x)
29 db1 = dz1
30 return dW1, db1, dw2, db2
state
forward pass
nodevaluehow
z6w · x = 2 · 3
r1z − y = 6 − 5
L1r² = 1²

Every node stores its output. That cache is the entire reason a single backward pass can produce every gradient — the alternative is recomputing the forward values once per parameter.

The scalar graph worked by hand above (w = 2, x = 3, y = 5 → ∂L/∂w = 6), then the identical procedure on the one-hidden-layer network. The weights are small explicit constants so every number is checkable, and they put one hidden unit at z₁ = −0.1: watch that unit get switched off by the ReLU and take 5 of the network's 11 parameter gradients to exactly zero with it — while ∂L/∂a₁ for the same unit stays non-zero, which is the distinction the gate exists to draw. The last two frames verify every gradient against a central finite difference (worst relative error 3.9e-11) and count what that verification cost: 22 forward passes against backprop's one.

What to notice:

  • The cache is the whole trick. Every local gradient is a function of a value the forward pass already computed — 2r2r needs rr, ∂L/∂w2\partial L/\partial \mathbf{w}_2 needs a1\mathbf{a}_1, ∂L/∂W1\partial L/\partial \mathbf{W}_1 needs x\mathbf{x}. Nothing is recomputed.
  • The BCE + sigmoid collapse is exact. Separately the two factors are −1.436-1.436 and 0.2110.211; their product is y^−y\hat{y} - y to the last digit. That is why frameworks fuse the two ops — the fused form also avoids dividing by y^(1−y^)→0\hat{y}(1-\hat{y}) \to 0.
  • Non-zero ∂L/∂a1\partial L/\partial \mathbf{a}_1, zero ∂L/∂z1\partial L/\partial \mathbf{z}_1. The trace's second hidden unit sits at z1=−0.1z_1 = -0.1. The loss is genuinely sensitive to its activation (0.1210.121), but the ReLU gate zeroes the gradient to its pre-activation, and those are different quantities. This is the mechanism behind dying ReLU: the gradient that would revive the unit is precisely the one being switched off.
  • One dead unit zeroes 5 of the 11 parameter gradients: the whole second row of W1\mathbf{W}_1, its bias, and the output weight reading from it.
  • The efficiency claim, counted. Verifying those 11 gradients by finite differences takes 22 forward passes; backprop produces all of them in one backward pass. At 10910^9 parameters that ratio is 2×109:12\times10^9 : 1.

Why backprop is efficient

Naively computing ∂L∂wi\frac{\partial L}{\partial w_i} by finite differences requires two forward passes per parameter. A network with millions of parameters would be intractable.

Backpropagation computes all gradients in one backward pass by reusing intermediate results. The key insight: gradients at earlier layers are computed by multiplying already-computed downstream gradients with local Jacobians. Total cost: ≈2\approx 2–3×3\times the forward pass cost (one pass to compute activations, one to propagate gradients, plus the weight gradient accumulation).

What the framework does with this

You never write the backward pass by hand, because PyTorch and JAX implement reverse-mode automatic differentiation: they build the computational graph on the fly as your forward pass runs, storing each operation's local derivative alongside its output, then replay that record in reverse.

Two details of that implementation are worth knowing, because both leak into APIs you use daily. The graph is replayed in reverse topological order, so a node's gradient is never read until every operation that consumed it has contributed. And contributions accumulate with += rather than overwrite, which is what makes a reused parameter come out right: a weight shared across 50 RNN timesteps is just a node with 50 outgoing edges, and the accumulation is the multivariable chain rule's sum over paths. It is also why optimizer.zero_grad() is your job rather than the framework's.

Deep dive
Automatic Differentiation: How Autograd Actually Works
Includes runnable notebook

Vanishing and exploding gradients

When many small Jacobians are multiplied together (deep networks, long sequences), the gradient can vanish:

∂L∂w1=∂L∂an⋅∏k=1n−1Jk\frac{\partial L}{\partial \mathbf{w}_1} = \frac{\partial L}{\partial \mathbf{a}_n} \cdot \prod_{k=1}^{n-1} J_k

If each ∥Jk∥<1\|J_k\| < 1, the gradient decays exponentially. Sigmoid's max derivative is 0.250.25 — stacking 20 sigmoid layers multiplies gradients by at most 0.2520≈10−120.25^{20} \approx 10^{-12}.

Solutions:

  • ReLU: gradient is exactly 1 for positive inputs, no saturation
  • Batch normalization: stabilizes activations, preventing saturation
  • Residual connections: add the input directly, creating gradient highways

Exercise

For h(x) = (3x + 1)⁴, what is h'(x)?

Need a hint?

Use the chain rule: d/dx f(g(x)) = f'(g(x)) · g'(x).

Exercise

In backpropagation, the gradient at each node is computed as the product of what two things?

Need a hint?

Each node has its own derivative and receives a signal from downstream.

Exercise

Why does stacking many sigmoid activations cause vanishing gradients?

Need a hint?

Think about what happens when you multiply many numbers less than 1.

Exercise

In a reverse-mode autodiff engine the backward pass ends with `parent.grad += local(node.grad)`. A colleague argues the `+=` should be `=`, since each node's gradient is computed exactly once. What breaks?

Need a hint?

What happens to a tensor that is used as an input by more than one operation?

Exercise

Why must a reverse-mode backward pass visit nodes in reverse topological order rather than any convenient order (say, the order they were created)?

Need a hint?

Think about a node whose output feeds two different downstream operations.

Exercise

During training, activation memory is usually the largest consumer of GPU memory — often larger than parameters, gradients and optimizer state combined. Which property of reverse-mode autodiff causes this?

Need a hint?

When is each operation's local derivative rule created, and what values does it need to close over?

Common mistakes

  1. Forgetting to cache forward-pass activations: backprop needs them to compute local gradients; without caching, you'd have to recompute the entire forward pass for every parameter.
  2. Confusing vanishing gradients with the chain rule being wrong: vanishing gradients are a consequence of multiplying many small terms (< 1) together, not an error in the chain rule itself.
  3. Applying the chain rule to matrix operations without respecting the transpose: ∂∂W(Wx)\frac{\partial}{\partial W}(Wx) involves x⊤x^\top, not xx; transposing preserves dimension consistency in the gradient.

Key takeaways

  • Backpropagation = chain rule applied to a computational graph.
  • Each node multiplies the upstream gradient by its local gradient and passes it back.
  • Backprop computes all gradients simultaneously in one pass. This is why neural network training is tractable.
  • Vanishing gradients happen when many <1< 1 factors multiply; solved by ReLU, batch norm, or residual connections.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

RelatedGo deeper
Chain Rule and BackpropagationDerivatives and GradientsGradient DescentRecurrent Neural NetworksJacobians & Vector Calcul…Layers & the Forward PassCNN Visualization & Adver…Automatic Differentiation…Backpropagation Through T…Logistic Regression: Grad…The Directional Derivative