Jacobians & Vector Calculus
Every layer in a neural network is a vector-valued function: it takes a vector input and produces a vector output. The gradient tools from earlier lessons (the derivative for scalars, the gradient for scalar-output functions) do not directly apply here. The right generalization is the Jacobian matrix.
Real-world examples: Backpropagation through a softmax layer, computing gradients through a batch-norm layer, understanding why narrow layers can bottleneck information flow, and analysing training stability all require Jacobian reasoning.
The Jacobian matrix
Let be a vector-valued function. Its Jacobian at a point is the matrix of all first-order partial derivatives:
Entry answers: how much does output change when input changes?
Special cases that you already know:
- : is a row vector — the gradient .
- : is an column vector — the ordinary derivative .
- : is a scalar — the familiar derivative .
The Jacobian linearly approximates near :
Think of as the best linear map from input perturbations to output perturbations.
Worked example: affine (linear) layer
The simplest layer is the affine map , where .
Output is . Taking the partial derivative:
So the Jacobian is exactly the weight matrix:
This is constant — it does not depend on . The affine map is its own linear approximation everywhere.
Shape intuition
The Jacobian always has shape (outputs × inputs). For a layer mapping inputs to outputs: . For a fully-connected layer with weight matrix , .
Worked example: element-wise function (ReLU)
For an element-wise function where the same scalar function is applied independently to each component, output only depends on input :
The Jacobian is a diagonal matrix with the pointwise derivatives on the diagonal:
For ReLU, (1 where active, 0 where inactive), so:
A neuron that is inactive (pre-activation ≤ 0) contributes a zero column and a zero row to the Jacobian — it passes no gradient signal during backpropagation. This is the dying ReLU problem viewed through the Jacobian.
Worked example: softmax Jacobian
The softmax function is defined by .
Outputs are coupled — changing changes every output simultaneously. The cross-partial derivatives are:
In matrix form, writing for the softmax output:
The outer-product term captures the coupling between outputs: increasing one logit raises one output probability but lowers all the others, because probabilities must sum to 1.
Numerical check
For a two-class softmax with logits :
The rows and columns each sum to zero — a necessary consequence of the probabilities summing to one.
import numpy as np
def softmax(x):
e = np.exp(x - x.max())
return e / e.sum()
def softmax_jacobian(x):
s = softmax(x)
return np.diag(s) - np.outer(s, s)
x = np.array([1.0, 0.0])
J = softmax_jacobian(x)
print("Softmax Jacobian:")
print(J.round(3))
print("Row sums:", J.sum(axis=1).round(8)) # all zero
The vector chain rule
When two vector-valued functions are composed, , the chain rule says the Jacobian of the composition is the product of Jacobians:
For a two-layer network with weight matrices , element-wise activation , and first-layer pre-activations (so is the activation's derivative at each unit):
This is the matrix product that backpropagation computes — the "credit assignment" chain.
Backprop = the chain rule on a graph
Forward (teal) computes values left → right. Backward (orange) multiplies local derivatives right → left: dy/dx = dy/du · du/dx = cos(u) · 2x. Each node just multiplies the incoming gradient by its own local derivative.
VJPs and backpropagation
Materializing the full Jacobian of a network means running one backward pass per output — of them, hopeless when is large. In ML we never need it. We need to compute
where is the upstream gradient (the gradient of the loss w.r.t. the output of the current layer). The VJP has the same shape as the input . It is the gradient of the loss w.r.t. .
Why VJPs instead of full Jacobians?
| Operation | Cost | What it computes |
|---|---|---|
| Full Jacobian | VJPs: the forward-pass cost | All partial derivatives |
| VJP | ≈ one forward pass | Gradient of loss w.r.t. inputs |
| JVP | ≈ one forward pass | Directional derivative of the outputs along |
Backpropagation is reverse-mode automatic differentiation: it computes VJPs from the output layer backwards. Forward-mode autodiff computes JVPs and is efficient when (few inputs, many outputs) — for example, in scientific simulation.
VJP of the affine layer
For , the Jacobian is . The VJP with upstream gradient is:
This is the backward pass formula you see in every backprop derivation: multiply by the transpose of the weight matrix.
Transposing in backprop
The transpose in is not an accident. It is the definition of the VJP. Forward pass: multiply by . Backward pass: multiply by . This follows directly from the Jacobian being .
Condition number and training stability
The condition number of the Jacobian (ratio of largest to smallest singular value) measures how much the layer stretches or squishes different directions of the input.
| Condition number | Effect on training |
|---|---|
| Well-conditioned; gradients flow evenly | |
| Ill-conditioned; some directions vanish, others explode | |
| Jacobian is near-singular; layer is an information bottleneck |
For a sequence of layers the condition number of the composition can grow exponentially — the deeper the network, the more critical initialization and normalization become. BatchNorm and LayerNorm both aim to keep the Jacobian of the normalized layer well-conditioned.
Vanishing and exploding gradients
The norm of the product of Jacobians grows or shrinks exponentially with depth. When each factor's spectral norm is below 1, the product vanishes; above 1, it explodes. Careful weight initialization (Xavier, He) targets spectral norm ≈ 1 at initialization.
Exercise
A neural network layer maps an input vector of size 256 to an output vector of size 128. What is the shape of its Jacobian?
Need a hint?
Rows index outputs; columns index inputs.
Exercise
For the affine layer y = Wx + b (W ∈ R^{m×n}), what is the Jacobian ∂y/∂x?
Need a hint?
Differentiate each output with respect to each input.
Exercise
During backpropagation through y = Wx + b, the upstream gradient is v = ∂L/∂y. What is the gradient ∂L/∂x passed to the previous layer?
Need a hint?
This is the vector-Jacobian product (VJP): v^T J.
Exercise
For f: ℝⁿ → ℝᵐ, forward-mode AD costs O(n) passes and reverse-mode costs O(m). Training a neural network has n ≈ 10⁹ parameters and m = 1 (a scalar loss). Which mode, and what is the price you pay for it?
Need a hint?
Reverse mode's advantage is not free — what does it have to keep around that forward mode does not?
Common mistakes
- Confusing Jacobian shape: the Jacobian of is , not . Rows index outputs, columns index inputs.
- Assuming diagonal Jacobians for all element-wise functions: this is true when each output depends on exactly one input. Softmax is element-wise in the sense of being a fixed formula per output, but outputs are coupled through the normalizer, making the Jacobian dense.
- Computing full Jacobians in code:
torch.autograd.functional.jacobianis useful for debugging but O(mn) in time. In production, only VJPs (the backward pass) are needed.
Key takeaways
- The Jacobian generalises the gradient to vector-valued functions.
- Affine layer: . Element-wise activation: . Softmax: .
- The vector chain rule chains Jacobians by matrix multiplication. This is what backpropagation implements.
- Backprop computes VJPs (not full Jacobians) — the upstream gradient times the Jacobian transpose.
- Condition number governs gradient flow: ill-conditioned Jacobians cause vanishing/exploding gradients.
Related concepts
- Chain Rule & Backpropagation: the scalar version of the same idea
- Multivariable Optimization: the Hessian, the second-order Jacobian
- SVD & Low-Rank Approximations: singular values determine the Jacobian's condition number