Derivatives and GradientsOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Derivatives and Gradients

Every gradient-based learning algorithm (from linear regression to GPT) reduces to one idea: find the direction in which a function decreases fastest, and take a small step in that direction. That direction is the negative gradient. This is the math of the loop's optimization stage (the search for parameters that minimize the objective) and this course is the toolkit that makes that search possible.

Deep dive
The ML Project Loop
Includes runnable notebook

Real-world examples: Every neural network trained with backpropagation, every optimizer (SGD, Adam), fine-tuning a language model — all rely on computing gradients via the chain rule.

The derivative is the slope of the tangent

xf(x)
x₀1.20
f(x₀)1.440
f′(x₀) slope2.400

f′(x₀) measures how fast f changes at x₀ — the slope of the yellow tangent line. Gradient descent uses exactly this slope to decide which way is downhill.

The derivative

The derivative of f(x)f(x) at a point measures how fast ff changes as xx changes:

f′(x)=dfdx=lim⁡h→0f(x+h)−f(x)hf'(x) = \frac{df}{dx} = \lim_{h \to 0} \frac{f(x+h) - f(x)}{h}

Geometrically, it's the slope of the tangent line at xx.

f′(x)f'(x)Meaning
>0> 0ff is increasing at xx
<0< 0ff is decreasing at xx
=0= 0Potential minimum, maximum, or saddle point

Common derivatives in ML

ddxxn=nxn−1,ddxex=ex,ddxln⁡x=1x\frac{d}{dx} x^n = nx^{n-1}, \quad \frac{d}{dx} e^x = e^x, \quad \frac{d}{dx} \ln x = \frac{1}{x}

ReLU derivative:

ddxmax⁡(0,x)={1x>00x<0\frac{d}{dx} \max(0, x) = \begin{cases} 1 & x > 0 \\ 0 & x < 0 \end{cases}

(At x=0x = 0 the derivative is undefined — the kink has no unique tangent. Frameworks simply use 00 there; any value in [0,1][0, 1] works in practice.)

Sigmoid derivative. With σ(x)=(1+e−x)−1\sigma(x) = (1 + e^{-x})^{-1}, apply the chain rule to the inner 1+e−x1 + e^{-x}:

σ′(x)=− (1+e−x)−2⋅(−e−x)=e−x(1+e−x)2\sigma'(x) = -\,(1 + e^{-x})^{-2}\cdot(-e^{-x}) = \frac{e^{-x}}{(1+e^{-x})^2}

Now split that fraction and recognize each piece as a sigmoid:

e−x(1+e−x)2=11+e−x⏟σ(x)⋅e−x1+e−x⏟1−σ(x)=σ(x) (1−σ(x))\frac{e^{-x}}{(1+e^{-x})^2} = \underbrace{\frac{1}{1+e^{-x}}}_{\sigma(x)}\cdot\underbrace{\frac{e^{-x}}{1+e^{-x}}}_{1-\sigma(x)} = \sigma(x)\,(1 - \sigma(x))

This compact form means a network can reuse the forward value σ(x)\sigma(x) to get its gradient for free. Numeric check: at x=0x=0, σ(0)=0.5\sigma(0)=0.5, so σ′(0)=0.5(1−0.5)=0.25\sigma'(0)=0.5(1-0.5)=0.25. At x=3x=3, σ(3)≈0.95\sigma(3)\approx0.95, so σ′(3)≈0.95(0.05)≈0.048\sigma'(3)\approx0.95(0.05)\approx0.048 — the gradient is nearly 5× smaller. That's saturation: near 0 or 1, one factor is tiny so σ′(x)→0\sigma'(x)\to0 and gradients die.

Partial derivatives

For a function of multiple variables f(x1,…,xn)f(x_1, \ldots, x_n), the partial derivative with respect to xix_i treats all other variables as constants:

∂f∂xi=lim⁡h→0f(x1,…,xi+h,…,xn)−f(…)h\frac{\partial f}{\partial x_i} = \lim_{h \to 0} \frac{f(x_1, \ldots, x_i + h, \ldots, x_n) - f(\ldots)}{h}

Example: f(x,y)=x2y+3yf(x, y) = x^2 y + 3y

∂f∂x=2xy,∂f∂y=x2+3\frac{\partial f}{\partial x} = 2xy, \qquad \frac{\partial f}{\partial y} = x^2 + 3

The gradient

The gradient collects all partial derivatives into one vector:

∇f(x)=[∂f∂x1∂f∂x2⋮∂f∂xn]\nabla f(\mathbf{x}) = \begin{bmatrix} \frac{\partial f}{\partial x_1} \\ \frac{\partial f}{\partial x_2} \\ \vdots \\ \frac{\partial f}{\partial x_n} \end{bmatrix}

The gradient points uphill

The gradient ∇f(x)\nabla f(\mathbf{x}) points in the direction of steepest ascent. Moving opposite to the gradient (−∇f)(-\nabla f) decreases ff as fast as possible. That's gradient descent.

Gradient descent in one equation

w←w−η∇wL(w)\mathbf{w} \leftarrow \mathbf{w} - \eta \nabla_{\mathbf{w}} \mathcal{L}(\mathbf{w})

Each parameter is nudged opposite to its partial derivative of the loss.

Computing gradients numerically

For checking analytical gradients (finite differences):

∂f∂xi≈f(x+ϵei)−f(x−ϵei)2ϵ\frac{\partial f}{\partial x_i} \approx \frac{f(\mathbf{x} + \epsilon \mathbf{e}_i) - f(\mathbf{x} - \epsilon \mathbf{e}_i)}{2\epsilon}
Python
import numpy as np

def f(x):
    return x[0]**2 + 2*x[1]**2 + x[0]*x[1]

def grad_f(x):
    # Analytical gradient
    return np.array([2*x[0] + x[1], 4*x[1] + x[0]])

def numerical_grad(f, x, eps=1e-5):
    grad = np.zeros_like(x)
    for i in range(len(x)):
        x_plus = x.copy(); x_plus[i] += eps
        x_minus = x.copy(); x_minus[i] -= eps
        grad[i] = (f(x_plus) - f(x_minus)) / (2 * eps)
    return grad

x = np.array([1.0, 2.0])
print("Analytical:", grad_f(x))   # [4., 9.]
print("Numerical: ", numerical_grad(f, x))  # ≈ [4., 9.]

Directional derivatives

The directional derivative measures the rate of change of ff along any unit direction d^\hat{\mathbf{d}}, and it is simply the dot product of the gradient with that direction:

Dd^f=∇f⋅d^=∥∇f∥cos⁡θD_{\hat{\mathbf{d}}} f = \nabla f \cdot \hat{\mathbf{d}} = \|\nabla f\| \cos\theta

This is maximized when d^=∇f∥∇f∥\hat{\mathbf{d}} = \frac{\nabla f}{\|\nabla f\|} — i.e., the gradient direction. This is why the gradient is the steepest ascent direction.

Numeric example: If ∇f(x)=[3,4]\nabla f(\mathbf{x}) = [3, 4] and we move in direction d^=[0.6,0.8]\hat{\mathbf{d}} = [0.6, 0.8] (a unit vector, since 0.62+0.82=10.6^2 + 0.8^2 = 1), then Dd^f=3(0.6)+4(0.8)=1.8+3.2=5.0D_{\hat{\mathbf{d}}} f = 3(0.6) + 4(0.8) = 1.8 + 3.2 = 5.0. Moving instead in d^=[0,1]\hat{\mathbf{d}} = [0, 1] gives Dd^f=4D_{\hat{\mathbf{d}}} f = 4 — less steep. The gradient direction [3,4]/5=[0.6,0.8][3,4]/5 = [0.6, 0.8] was already the unit vector that achieves the maximum rate of ∥∇f∥=5\|\nabla f\| = 5.

Gradient magnitude and loss landscapes

When training neural networks, very large gradient magnitudes indicate steep curvature (exploding gradients). Very small gradients indicate flat regions (vanishing gradients or near-optimal parameters). Gradient clipping and careful initialization manage both.

For the full derivation — from the limit definition through the chain rule to the dot-product formula, and a proof that the gradient is steepest ascent — see the deep dive:

Deep dive
The Directional Derivative
Includes runnable notebook

Exercise

For f(x, y) = 3x²y + y², what is ∂f/∂x?

Need a hint?

Treat y as a constant and differentiate with respect to x.

Exercise

The gradient ∇f(x) points in the direction of steepest ___.

Need a hint?

Think about which direction gradient descent moves.

Exercise

The derivative of sigmoid σ(x) is σ(x)(1 − σ(x)). What is its maximum value?

Need a hint?

Maximize σ(x)(1 − σ(x)) — treat it as a function of p = σ(x) in [0,1].

Common mistakes

  1. Confusing the gradient direction with steepest descent: the gradient points uphill; descent follows the negative gradient.
  2. Feeding an unnormalized direction into the directional derivative: Dd^f=∇f⋅d^D_{\hat{\mathbf{d}}}f = \nabla f \cdot \hat{\mathbf{d}} is defined for unit vectors; using a raw v\mathbf{v} silently scales the answer by ∥v∥\|\mathbf{v}\|.
  3. Forgetting that the gradient of a scalar function with respect to a vector is itself a vector of the same shape as the input.

Key takeaways

  • The derivative is the rate of change; for multivariable functions, the gradient is the vector of partial derivatives.
  • The gradient points uphill — gradient descent moves in the negative gradient direction.
  • Setting the gradient to zero finds critical points — candidates for minima, maxima, or saddle points.
  • The directional derivative is the dot product of the gradient with any unit direction.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

RelatedGo deeper
Derivatives and GradientsEigenvalues and Eigenvect…Chain Rule and Backpropag…Gradient DescentMultivariable OptimizationThe ML Project LoopThe Directional Derivative