Jacobians & Vector CalculusOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Jacobians & Vector Calculus

Every layer in a neural network is a vector-valued function: it takes a vector input and produces a vector output. The gradient tools from earlier lessons (the derivative for scalars, the gradient for scalar-output functions) do not directly apply here. The right generalization is the Jacobian matrix.

Real-world examples: Backpropagation through a softmax layer, computing gradients through a batch-norm layer, understanding why narrow layers can bottleneck information flow, and analysing training stability all require Jacobian reasoning.

The Jacobian matrix

Let f:Rn→Rm\mathbf{f}: \mathbb{R}^n \to \mathbb{R}^m be a vector-valued function. Its Jacobian at a point x\mathbf{x} is the m×nm \times n matrix of all first-order partial derivatives:

Jf(x)=[∂f1∂x1⋯∂f1∂xn⋮⋱⋮∂fm∂x1⋯∂fm∂xn]\mathbf{J}_{\mathbf{f}}(\mathbf{x}) = \begin{bmatrix} \dfrac{\partial f_1}{\partial x_1} & \cdots & \dfrac{\partial f_1}{\partial x_n} \\[6pt] \vdots & \ddots & \vdots \\[4pt] \dfrac{\partial f_m}{\partial x_1} & \cdots & \dfrac{\partial f_m}{\partial x_n} \end{bmatrix}

Entry (i,j)(i, j) answers: how much does output ii change when input jj changes?

Special cases that you already know:

  • m=1m = 1: J\mathbf{J} is a 1×n1 \times n row vector — the gradient ∇f⊤\nabla f^\top.
  • n=1n = 1: J\mathbf{J} is an m×1m \times 1 column vector — the ordinary derivative df/dxd\mathbf{f}/dx.
  • m=n=1m = n = 1: J\mathbf{J} is a scalar — the familiar derivative f′(x)f'(x).

The Jacobian linearly approximates f\mathbf{f} near x0\mathbf{x}_0:

f(x0+δ)≈f(x0)+J δ\mathbf{f}(\mathbf{x}_0 + \boldsymbol{\delta}) \approx \mathbf{f}(\mathbf{x}_0) + \mathbf{J}\,\boldsymbol{\delta}

Think of J\mathbf{J} as the best linear map from input perturbations to output perturbations.

Worked example: affine (linear) layer

The simplest layer is the affine map f(x)=Wx+b\mathbf{f}(\mathbf{x}) = \mathbf{W}\mathbf{x} + \mathbf{b}, where W∈Rm×n\mathbf{W} \in \mathbb{R}^{m \times n}.

Output ii is fi(x)=∑kWikxk+bif_i(\mathbf{x}) = \sum_k W_{ik} x_k + b_i. Taking the partial derivative:

∂fi∂xj=Wij\frac{\partial f_i}{\partial x_j} = W_{ij}

So the Jacobian is exactly the weight matrix:

Jf=W∈Rm×n\mathbf{J}_{\mathbf{f}} = \mathbf{W} \quad \in \mathbb{R}^{m \times n}

This is constant — it does not depend on x\mathbf{x}. The affine map is its own linear approximation everywhere.

Shape intuition

The Jacobian always has shape (outputs × inputs). For a layer mapping nn inputs to mm outputs: J∈Rm×n\mathbf{J} \in \mathbb{R}^{m \times n}. For a fully-connected layer with weight matrix Wm×n\mathbf{W}^{m \times n}, J=W\mathbf{J} = \mathbf{W}.

Worked example: element-wise function (ReLU)

For an element-wise function f(x)=(f(x1),…,f(xn))\mathbf{f}(\mathbf{x}) = (f(x_1), \ldots, f(x_n)) where the same scalar function ff is applied independently to each component, output ii only depends on input ii:

∂fi∂xj=f′(xi)⋅1[i=j]\frac{\partial f_i}{\partial x_j} = f'(x_i) \cdot \mathbb{1}[i = j]

The Jacobian is a diagonal matrix with the pointwise derivatives on the diagonal:

Jf(x)=diag⁡(f′(x1),…,f′(xn))\mathbf{J}_{\mathbf{f}}(\mathbf{x}) = \operatorname{diag}(f'(x_1), \ldots, f'(x_n))

For ReLU, f′(xi)=1[xi>0]f'(x_i) = \mathbb{1}[x_i > 0] (1 where active, 0 where inactive), so:

JReLU(x)=diag⁡ ⁣(1[x1>0], …, 1[xn>0])\mathbf{J}_{\text{ReLU}}(\mathbf{x}) = \operatorname{diag}\!\big(\mathbb{1}[x_1 > 0],\ \ldots,\ \mathbb{1}[x_n > 0]\big)

A neuron that is inactive (pre-activation ≤ 0) contributes a zero column and a zero row to the Jacobian — it passes no gradient signal during backpropagation. This is the dying ReLU problem viewed through the Jacobian.

Worked example: softmax Jacobian

The softmax function σ:Rn→Rn\sigma: \mathbb{R}^n \to \mathbb{R}^n is defined by σi(x)=exi/∑kexk\sigma_i(\mathbf{x}) = e^{x_i} / \sum_k e^{x_k}.

Outputs are coupled — changing xjx_j changes every output simultaneously. The cross-partial derivatives are:

∂σi∂xj={σi(1−σi)i=j(diagonal)−σiσji≠j(off-diagonal)\frac{\partial \sigma_i}{\partial x_j} = \begin{cases} \sigma_i(1 - \sigma_i) & i = j \quad \text{(diagonal)} \\ -\sigma_i \sigma_j & i \neq j \quad \text{(off-diagonal)} \end{cases}

In matrix form, writing σ\boldsymbol{\sigma} for the softmax output:

Jσ(x)=diag⁡(σ)−σσ⊤\mathbf{J}_{\sigma}(\mathbf{x}) = \operatorname{diag}(\boldsymbol{\sigma}) - \boldsymbol{\sigma}\boldsymbol{\sigma}^\top

The outer-product term σσ⊤\boldsymbol{\sigma}\boldsymbol{\sigma}^\top captures the coupling between outputs: increasing one logit raises one output probability but lowers all the others, because probabilities must sum to 1.

Numerical check

For a two-class softmax with logits (1,0)(1, 0):

σ=1e+1[e1]≈[0.7310.269]\boldsymbol{\sigma} = \frac{1}{e + 1}\begin{bmatrix} e \\ 1 \end{bmatrix} \approx \begin{bmatrix} 0.731 \\ 0.269 \end{bmatrix} J=[0.731000.269]−[0.7310.269][0.7310.269]=[0.197−0.197−0.1970.197]\mathbf{J} = \begin{bmatrix} 0.731 & 0 \\ 0 & 0.269 \end{bmatrix} - \begin{bmatrix} 0.731 \\ 0.269 \end{bmatrix} \begin{bmatrix} 0.731 & 0.269 \end{bmatrix} = \begin{bmatrix} 0.197 & -0.197 \\ -0.197 & 0.197 \end{bmatrix}

The rows and columns each sum to zero — a necessary consequence of the probabilities summing to one.

Python
import numpy as np

def softmax(x):
    e = np.exp(x - x.max())
    return e / e.sum()

def softmax_jacobian(x):
    s = softmax(x)
    return np.diag(s) - np.outer(s, s)

x = np.array([1.0, 0.0])
J = softmax_jacobian(x)
print("Softmax Jacobian:")
print(J.round(3))
print("Row sums:", J.sum(axis=1).round(8))  # all zero

The vector chain rule

When two vector-valued functions are composed, h=g∘f\mathbf{h} = \mathbf{g} \circ \mathbf{f}, the chain rule says the Jacobian of the composition is the product of Jacobians:

Jh(x)=Jg(f(x))⋅Jf(x)\mathbf{J}_{\mathbf{h}}(\mathbf{x}) = \mathbf{J}_{\mathbf{g}}(\mathbf{f}(\mathbf{x})) \cdot \mathbf{J}_{\mathbf{f}}(\mathbf{x})

For a two-layer network with weight matrices W1,W2\mathbf{W}_1, \mathbf{W}_2, element-wise activation ϕ\phi, and first-layer pre-activations z1=W1x+b1\mathbf{z}_1 = \mathbf{W}_1\mathbf{x} + \mathbf{b}_1 (so ϕ′(z1)\phi'(\mathbf{z}_1) is the activation's derivative at each unit):

Jnet(x)=W2⋅diag⁡(ϕ′(z1))⋅W1\mathbf{J}_{\text{net}}(\mathbf{x}) = \mathbf{W}_2 \cdot \operatorname{diag}(\phi'(\mathbf{z}_1)) \cdot \mathbf{W}_1

This is the matrix product that backpropagation computes — the "credit assignment" chain.

Backprop = the chain rule on a graph

squaresinx1.50u = x²2.25y = sin u0.778node values (forward pass)
forward y0.778
gradient dy/dx-1.885

Forward (teal) computes values left → right. Backward (orange) multiplies local derivatives right → left: dy/dx = dy/du · du/dx = cos(u) · 2x. Each node just multiplies the incoming gradient by its own local derivative.

VJPs and backpropagation

Materializing the full m×nm \times n Jacobian of a network means running one backward pass per output — mm of them, hopeless when mm is large. In ML we never need it. We need to compute

v⊤J(a vector-Jacobian product, or VJP)\mathbf{v}^\top \mathbf{J} \quad \text{(a vector-Jacobian product, or VJP)}

where v\mathbf{v} is the upstream gradient (the gradient of the loss w.r.t. the output of the current layer). The VJP has the same shape as the input x\mathbf{x}. It is the gradient of the loss w.r.t. x\mathbf{x}.

Why VJPs instead of full Jacobians?

OperationCostWhat it computes
Full Jacobianmm VJPs: m×m\times the forward-pass costAll partial derivatives
VJP v⊤J\mathbf{v}^\top \mathbf{J}≈ one forward passGradient of loss w.r.t. inputs
JVP Ju\mathbf{J}\mathbf{u}≈ one forward passDirectional derivative of the outputs along u\mathbf{u}

Backpropagation is reverse-mode automatic differentiation: it computes VJPs from the output layer backwards. Forward-mode autodiff computes JVPs and is efficient when n≪mn \ll m (few inputs, many outputs) — for example, in scientific simulation.

VJP of the affine layer

For y=Wx+b\mathbf{y} = \mathbf{W}\mathbf{x} + \mathbf{b}, the Jacobian is W\mathbf{W}. The VJP with upstream gradient v∈Rm\mathbf{v} \in \mathbb{R}^m is:

v⊤J=v⊤W⟹∂L∂x=W⊤v\mathbf{v}^\top \mathbf{J} = \mathbf{v}^\top \mathbf{W} \quad \Longrightarrow \quad \frac{\partial \mathcal{L}}{\partial \mathbf{x}} = \mathbf{W}^\top \mathbf{v}

This is the backward pass formula you see in every backprop derivation: multiply by the transpose of the weight matrix.

Transposing in backprop

The transpose in W⊤v\mathbf{W}^\top \mathbf{v} is not an accident. It is the definition of the VJP. Forward pass: multiply by W\mathbf{W}. Backward pass: multiply by W⊤\mathbf{W}^\top. This follows directly from the Jacobian being W\mathbf{W}.

Condition number and training stability

The condition number of the Jacobian κ(J)=σmax⁡/σmin⁡\kappa(\mathbf{J}) = \sigma_{\max} / \sigma_{\min} (ratio of largest to smallest singular value) measures how much the layer stretches or squishes different directions of the input.

Condition numberEffect on training
κ≈1\kappa \approx 1Well-conditioned; gradients flow evenly
κ≫1\kappa \gg 1Ill-conditioned; some directions vanish, others explode
κ→∞\kappa \to \inftyJacobian is near-singular; layer is an information bottleneck

For a sequence of layers the condition number of the composition can grow exponentially — the deeper the network, the more critical initialization and normalization become. BatchNorm and LayerNorm both aim to keep the Jacobian of the normalized layer well-conditioned.

Vanishing and exploding gradients

The norm of the product of Jacobians ∥JL⋯J1∥\|\mathbf{J}_L \cdots \mathbf{J}_1\| grows or shrinks exponentially with depth. When each factor's spectral norm is below 1, the product vanishes; above 1, it explodes. Careful weight initialization (Xavier, He) targets spectral norm ≈ 1 at initialization.

Exercise

A neural network layer maps an input vector of size 256 to an output vector of size 128. What is the shape of its Jacobian?

Need a hint?

Rows index outputs; columns index inputs.

Exercise

For the affine layer y = Wx + b (W ∈ R^{m×n}), what is the Jacobian ∂y/∂x?

Need a hint?

Differentiate each output with respect to each input.

Exercise

During backpropagation through y = Wx + b, the upstream gradient is v = ∂L/∂y. What is the gradient ∂L/∂x passed to the previous layer?

Need a hint?

This is the vector-Jacobian product (VJP): v^T J.

Exercise

For f: ℝⁿ → ℝᵐ, forward-mode AD costs O(n) passes and reverse-mode costs O(m). Training a neural network has n ≈ 10⁹ parameters and m = 1 (a scalar loss). Which mode, and what is the price you pay for it?

Need a hint?

Reverse mode's advantage is not free — what does it have to keep around that forward mode does not?

Common mistakes

  1. Confusing Jacobian shape: the Jacobian of f:Rn→Rm\mathbf{f}: \mathbb{R}^n \to \mathbb{R}^m is m×nm \times n, not n×mn \times m. Rows index outputs, columns index inputs.
  2. Assuming diagonal Jacobians for all element-wise functions: this is true when each output depends on exactly one input. Softmax is element-wise in the sense of being a fixed formula per output, but outputs are coupled through the normalizer, making the Jacobian dense.
  3. Computing full Jacobians in code: torch.autograd.functional.jacobian is useful for debugging but O(mn) in time. In production, only VJPs (the backward pass) are needed.

Key takeaways

  • The Jacobian J∈Rm×n\mathbf{J} \in \mathbb{R}^{m \times n} generalises the gradient to vector-valued functions.
  • Affine layer: J=W\mathbf{J} = \mathbf{W}. Element-wise activation: J=diag⁡(ϕ′(x))\mathbf{J} = \operatorname{diag}(\phi'(\mathbf{x})). Softmax: J=diag⁡(σ)−σσ⊤\mathbf{J} = \operatorname{diag}(\boldsymbol{\sigma}) - \boldsymbol{\sigma}\boldsymbol{\sigma}^\top.
  • The vector chain rule chains Jacobians by matrix multiplication. This is what backpropagation implements.
  • Backprop computes VJPs (not full Jacobians) — the upstream gradient times the Jacobian transpose.
  • Condition number κ(J)\kappa(\mathbf{J}) governs gradient flow: ill-conditioned Jacobians cause vanishing/exploding gradients.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

RelatedGo deeper
Jacobians & Vector CalculusChain Rule and Backpropag…Multivariable OptimizationSVD & Low-Rank Approximat…Automatic Differentiation…