Derivatives and Gradients
Every gradient-based learning algorithm (from linear regression to GPT) reduces to one idea: find the direction in which a function decreases fastest, and take a small step in that direction. That direction is the negative gradient. This is the math of the loop's optimization stage (the search for parameters that minimize the objective) and this course is the toolkit that makes that search possible.
Real-world examples: Every neural network trained with backpropagation, every optimizer (SGD, Adam), fine-tuning a language model — all rely on computing gradients via the chain rule.
The derivative is the slope of the tangent
f′(x₀) measures how fast f changes at x₀ — the slope of the yellow tangent line. Gradient descent uses exactly this slope to decide which way is downhill.
The derivative
The derivative of at a point measures how fast changes as changes:
Geometrically, it's the slope of the tangent line at .
| Meaning | |
|---|---|
| is increasing at | |
| is decreasing at | |
| Potential minimum, maximum, or saddle point |
Common derivatives in ML
ReLU derivative:
(At the derivative is undefined — the kink has no unique tangent. Frameworks simply use there; any value in works in practice.)
Sigmoid derivative. With , apply the chain rule to the inner :
Now split that fraction and recognize each piece as a sigmoid:
This compact form means a network can reuse the forward value to get its gradient for free. Numeric check: at , , so . At , , so — the gradient is nearly 5× smaller. That's saturation: near 0 or 1, one factor is tiny so and gradients die.
Partial derivatives
For a function of multiple variables , the partial derivative with respect to treats all other variables as constants:
Example:
The gradient
The gradient collects all partial derivatives into one vector:
The gradient points uphill
The gradient points in the direction of steepest ascent. Moving opposite to the gradient decreases as fast as possible. That's gradient descent.
Gradient descent in one equation
Each parameter is nudged opposite to its partial derivative of the loss.
Computing gradients numerically
For checking analytical gradients (finite differences):
import numpy as np
def f(x):
return x[0]**2 + 2*x[1]**2 + x[0]*x[1]
def grad_f(x):
# Analytical gradient
return np.array([2*x[0] + x[1], 4*x[1] + x[0]])
def numerical_grad(f, x, eps=1e-5):
grad = np.zeros_like(x)
for i in range(len(x)):
x_plus = x.copy(); x_plus[i] += eps
x_minus = x.copy(); x_minus[i] -= eps
grad[i] = (f(x_plus) - f(x_minus)) / (2 * eps)
return grad
x = np.array([1.0, 2.0])
print("Analytical:", grad_f(x)) # [4., 9.]
print("Numerical: ", numerical_grad(f, x)) # ≈ [4., 9.]
Directional derivatives
The directional derivative measures the rate of change of along any unit direction , and it is simply the dot product of the gradient with that direction:
This is maximized when — i.e., the gradient direction. This is why the gradient is the steepest ascent direction.
Numeric example: If and we move in direction (a unit vector, since ), then . Moving instead in gives — less steep. The gradient direction was already the unit vector that achieves the maximum rate of .
Gradient magnitude and loss landscapes
When training neural networks, very large gradient magnitudes indicate steep curvature (exploding gradients). Very small gradients indicate flat regions (vanishing gradients or near-optimal parameters). Gradient clipping and careful initialization manage both.
For the full derivation — from the limit definition through the chain rule to the dot-product formula, and a proof that the gradient is steepest ascent — see the deep dive:
Exercise
For f(x, y) = 3x²y + y², what is ∂f/∂x?
Need a hint?
Treat y as a constant and differentiate with respect to x.
Exercise
The gradient ∇f(x) points in the direction of steepest ___.
Need a hint?
Think about which direction gradient descent moves.
Exercise
The derivative of sigmoid σ(x) is σ(x)(1 − σ(x)). What is its maximum value?
Need a hint?
Maximize σ(x)(1 − σ(x)) — treat it as a function of p = σ(x) in [0,1].
Common mistakes
- Confusing the gradient direction with steepest descent: the gradient points uphill; descent follows the negative gradient.
- Feeding an unnormalized direction into the directional derivative: is defined for unit vectors; using a raw silently scales the answer by .
- Forgetting that the gradient of a scalar function with respect to a vector is itself a vector of the same shape as the input.
Key takeaways
- The derivative is the rate of change; for multivariable functions, the gradient is the vector of partial derivatives.
- The gradient points uphill — gradient descent moves in the negative gradient direction.
- Setting the gradient to zero finds critical points — candidates for minima, maxima, or saddle points.
- The directional derivative is the dot product of the gradient with any unit direction.
Related concepts
- Chain Rule and Backpropagation: computing gradients through composed functions
- Gradient Descent: using gradients to train models
- Multivariable Optimization: second-order methods and curvature