Automatic Differentiation: How Autograd Actually WorksOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Automatic Differentiation: How Autograd Actually Works

Most explanations of backpropagation stop at the chain rule, which is the mathematics. This page is about the implementation: the data structure PyTorch, JAX and TensorFlow build while your forward pass runs, and the algorithm they run over it when you call .backward(). It sits in the optimization slot of the ML loop, and what it replaced was writing derivatives by hand for every architecture you ever wanted to try.

The gap is worth naming, because "backprop is just the chain rule" leaves you unable to answer the questions that actually come up: why activations dominate your memory bill, why zero_grad() exists, why a tensor used twice needs no special handling, and what retain_graph=True is retaining.

Four ways to get a derivative

Autodiff is neither symbolic differentiation nor numerical differentiation, and the distinction is the entire reason it works at scale.

ApproachHowCost for ∇f\nabla f, f:Rn→Rf:\mathbb{R}^n\to\mathbb{R}Problem
SymbolicManipulate the expression algebraically—Expression swell: the derivative of a deep composition is exponentially large to write down, and control flow has no symbolic form
Numericalf(x+h)−f(x−h)2h\dfrac{f(x+h)-f(x-h)}{2h} per input2n2n forward passesToo slow (n≈109n \approx 10^9), and truncation vs. round-off means you never get more than ~8 digits
Forward-mode ADPropagate derivatives alongside valuesnn forward passesCost scales with inputs
Reverse-mode ADRecord values forward, propagate derivatives backward1 forward + 1 backwardMust store the intermediates

The last row is the one deep learning runs on. The cost of reverse mode is independent of the number of parameters, so a single backward pass produces gradients for all 10910^9 of them at roughly 2× the cost of the forward pass.

The rule for choosing a mode

For f:Rn→Rmf: \mathbb{R}^n \to \mathbb{R}^m, forward mode costs O(n)O(n) passes and reverse mode costs O(m)O(m). Training has nn = millions of parameters and m=1m = 1 (a scalar loss), so reverse mode wins by a factor of nn. Flip the shape — one input, many outputs, as in sensitivity analysis or Jacobian-vector products — and forward mode is the right tool. JAX exposes both directly as jax.grad / jax.jvp, which is why its API distinguishes jvp from vjp where PyTorch mostly just says backward.

The tape

A framework never sees your formula. It sees a sequence of operation calls, and it writes each one down as it happens. That record — the tape, or the autograd graph — is built as a side effect of the forward pass.

Each recorded node stores three things:

  1. The output value, which the next operation needs.
  2. References to its input nodes, which are the edges of the graph.
  3. A closure that turns an incoming gradient into gradients for those inputs — the local derivative, with whatever forward values it needs captured at record time.

Point 3 is where activation memory comes from. mul(a, b) has to remember a and b to compute its backward, because ∂(ab)/∂a=b\partial(ab)/\partial a = b. Every op makes its own choice about what to keep: tanh keeps its output (since 1−t21 - t^2 is cheapest from tt), ReLU keeps only a sign mask, matmul keeps both operands. Those decisions, summed over every layer, are usually a larger number than your parameter count.

Reverse-mode autodiff: building the tape and replaying it backwards

1/12

The expression, and why this one. a = w·x, h = tanh(a), o = w·h, d = o − t, L = d·d, with w = 0.7, x = 1.5, t = 1. Look at where **w** appears: once at the very bottom feeding the tanh, and again near the top multiplying its output. One parameter, two uses. That is weight sharing — the same structure an RNN has across timesteps — and it is precisely the case where "just apply the chain rule" stops being a complete instruction, because w's gradient is a sum over two paths of different lengths.

python
1class Node:
2 def __init__(self, v, src=()):
3 self.v = v # forward value
4 self.g = 0.0 # accumulated grad
5 self.src = src # (parent, local grad)
6
7def mul(a, b):
8 # locals captured at RECORD time
9 return Node(a.v * b.v, (
10 (a, lambda g: g * b.v),
11 (b, lambda g: g * a.v)))
12
13def sub(a, b):
14 return Node(a.v - b.v, (
15 (a, lambda g: g),
16 (b, lambda g: -g)))
17
18def tanh(a):
19 t = math.tanh(a.v)
20 return Node(t, (
21 (a, lambda g: g * (1 - t * t)),))
22
23def toposort(out):
24 order, seen = [], set()
25 def visit(n):
26 if id(n) in seen: return
27 seen.add(id(n))
28 for p, _ in n.src:
29 visit(p)
30 order.append(n) # parents first
31 visit(out)
32 return order
33
34def backward(out):
35 order = toposort(out)
36 out.g = 1.0
37 for n in reversed(order):
38 for parent, local in n.src:
39 parent.g += local(n.g)
state
inputs
w → 0.7x → 1.5t → 1

A framework does not read your expression. It has no idea a formula exists — it only sees a sequence of operations being called, and it writes each one down as it happens. That written record is the tape.

A real micro-autograd, recorded as it runs. The expression uses w twice (a = w·x and o = w·h) so it is a genuine DAG rather than a chain, which is what makes the two structural pieces necessary: the topological sort, so a node's gradient is never read before all of its consumers have written to it, and the `+=`, so both paths add instead of one overwriting the other. The last frame changes that `+=` to `=`, re-runs the traversal, and checks both against a finite difference — the bug returns 17.2% of the true gradient with no error of any kind, and the frame breaks down why that is even worse than w's own two paths would explain.

What to notice as you step through it:

  • The local derivatives are stored during the forward pass, not computed during the backward one. The backward pass is a replay, not a derivation.
  • w is used twice, and nothing in the recording treats that as special. The tape just gets a second edge into the same node.
  • The topological sort is what makes the second edge safe. A node's gradient is not final until every consumer has pushed into it.
  • parent.g += ... is load-bearing. The final frame changes it to = and gets 34.3% of the correct gradient back, with no error raised.

Why the order matters

The backward pass cannot run in any order. Consider the fan-out from the trace: w feeds both a = w·x and o = w·h. Its true gradient is a sum over paths:

∂L∂w=∂L∂o∂o∂w⏟short path+∂L∂a∂a∂w⏟long path\frac{\partial L}{\partial w} = \underbrace{\frac{\partial L}{\partial o}\frac{\partial o}{\partial w}}_{\text{short path}} + \underbrace{\frac{\partial L}{\partial a}\frac{\partial a}{\partial w}}_{\text{long path}}

If you read w.grad after the short path has contributed but before the long one has, you get a number that is well-formed, plausible, and wrong. The invariant a correct implementation must maintain is:

Never process a node until every node that consumes its output has already been processed.

That is exactly reverse topological order. PyTorch enforces it with a dependency counter — each node knows how many consumers still owe it a contribution, and enters the ready queue when that count hits zero — rather than a depth-first sort, but the guarantee is the same.

The multivariable chain rule is where the sum comes from: for LL depending on ww through several intermediates uiu_i,

∂L∂w=∑i∂L∂ui∂ui∂w\frac{\partial L}{\partial w} = \sum_i \frac{\partial L}{\partial u_i}\frac{\partial u_i}{\partial w}

Accumulation is that sum, implemented. This is why weight sharing needs no special support: an RNN reusing WhW_h across 50 timesteps, or a convolution reusing a kernel across every spatial position, is just a node with 50 (or 50,000) outgoing edges, and += handles it identically to two.

Define-by-run vs. define-and-run

Two designs, and the trade-off between them explains most of the last decade of framework churn.

Define-by-run (dynamic)Define-and-run (static)
When the graph is builtWhile the forward pass executes, every iterationOnce, ahead of time
ExamplesPyTorch eager, TF2 eager, Chainer (the original)TensorFlow 1.x, torch.compile, jax.jit, ONNX
Control flowOrdinary Python if/while — the tape records whichever branch ranNeeds graph-level primitives (tf.cond, lax.scan) or tracing that bakes one branch in
DebuggingStandard stack traces and breakpointsErrors surface at graph-execution time, far from the code that caused them
OptimizationLimited: the framework sees one op at a timeWhole-graph: operator fusion, memory planning, dead-code elimination, kernel selection

Dynamic won on ergonomics, and then the industry got the optimization back by tracing dynamic code into a static graph (torch.compile, jax.jit). That is also where tracing's sharp edge comes from: a traced function records the branch taken for the shapes and values it was traced with. Python control flow that depends on tensor values is silently frozen into the trace, which is why JAX makes you write lax.cond and why torch.compile inserts guards and re-compiles when they fail.

What it costs, and what the API exposes

Every piece of autograd's bookkeeping shows up somewhere in the API you already use.

  • Activation memory. The tape holds forward values alive until the backward pass consumes them, so memory scales with depth × batch × width, not with parameter count. This is what
    Deep dive
    Gradient Checkpointing
    Includes runnable notebook
    trades away, storing a sparse set of checkpoints and recomputing the rest.
  • loss.backward() frees the graph by default. Once the buffers have been consumed they are released, which is why a second backward() on the same graph raises an error and why retain_graph=True exists. You need it when one forward pass feeds two losses backwarded separately, as in some GAN and meta-learning setups.
  • optimizer.zero_grad() is required because of +=. The framework's accumulation never resets on its own: it cannot know where one backward pass ends and the next begins. That is a feature — it is how you implement gradient accumulation across micro-batches to simulate a larger batch — but forget it and you are summing this step's gradient onto last step's.
  • torch.no_grad() and .detach() stop the recording. The first suppresses tape-building entirely (inference, and the optimizer's own parameter updates); the second cuts one tensor out of the graph, which is how you stop a gradient flowing into a target network or a frozen encoder.
  • Only leaf tensors keep .grad. Intermediates get their gradients passed along and then dropped, because keeping them all would double an already-large memory bill. retain_grad() opts a specific intermediate back in.
  • In-place operations can corrupt the backward pass. If an op stored a tensor for its backward and you overwrite that tensor in place, the replay uses the wrong number. PyTorch version-counts tensors and raises rather than returning a silently wrong gradient.

Autodiff is exact, but only of the code you wrote

Reverse-mode AD is not an approximation — it applies exact derivative rules, accurate to floating-point round-off, which is why it beats finite differences on accuracy as well as speed. But it differentiates the program that ran, and where your program is not differentiable it will still hand you a number. relu at exactly 0, abs at 0, and every round/floor/argmax returns a convenient subgradient (usually 0) with no warning. Straight-through estimators and the Gumbel-softmax trick exist precisely to put a useful fiction there instead of the default one.

Building intuition: implement it once

The notebook for this page builds a working reverse-mode autodiff engine in about 60 lines of NumPy — Value objects, operator overloading, topological sort, accumulation — trains a small MLP with it, and checks every gradient against torch.autograd. It is the single most useful hour available for making frameworks stop feeling like magic, and it is also the fastest way to internalize why the answer to "what happens in the backward pass?" is a data-structure answer rather than a calculus one.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

RelatedReferenced by
Automatic Differentiation: How Au…Gradient CheckpointingBackpropagation Through T…Vanishing & Exploding Gra…Chain Rule and Backpropag…Jacobians & Vector Calcul…Layers & the Forward Pass