The Directional DerivativeOpen in Colab

The Directional Derivative

A partial derivative answers a narrow question: how fast does ff change if I nudge one coordinate, holding the rest fixed? The directional derivative answers the general one: how fast does ff change if I step in an arbitrary direction u\mathbf{u}? Partial derivatives are just the special case where u\mathbf{u} is a coordinate axis. This single quantity is what makes "the gradient points in the direction of steepest ascent" a theorem rather than a slogan.

The limit definition of the directional derivative

Fix a point x∈Rn\mathbf{x} \in \mathbb{R}^n and a unit direction u^\hat{\mathbf{u}} (∥u^∥=1\|\hat{\mathbf{u}}\| = 1). Slide along the ray x+h u^\mathbf{x} + h\,\hat{\mathbf{u}} and measure the per-unit-distance change:

Du^f(x)=lim⁡h→0f(x+h u^)−f(x)hD_{\hat{\mathbf{u}}} f(\mathbf{x}) = \lim_{h \to 0} \frac{f(\mathbf{x} + h\,\hat{\mathbf{u}}) - f(\mathbf{x})}{h}

Setting u^=ei\hat{\mathbf{u}} = \mathbf{e}_i (the ii-th standard basis vector) recovers the partial derivative ∂f/∂xi\partial f / \partial x_i exactly, so partials are directional derivatives along the axes.

The unit-vector requirement is not optional

Du^fD_{\hat{\mathbf{u}}} f is a rate per unit distance. If u\mathbf{u} is not normalized, the limit scales by ∥u∥\|\mathbf{u}\| and you get a rate per unit of hh instead — an apples-to-oranges quantity. Always divide by ∥u∥\|\mathbf{u}\| first: u^=u/∥u∥\hat{\mathbf{u}} = \mathbf{u}/\|\mathbf{u}\|.

Reduction to the dot product with the gradient

Define the single-variable restriction g(h)=f(x+h u^)g(h) = f(\mathbf{x} + h\,\hat{\mathbf{u}}). By the limit definition, Du^f(x)=g′(0)D_{\hat{\mathbf{u}}} f(\mathbf{x}) = g'(0). Now differentiate gg with the multivariable chain rule. Writing the path as γ(h)=x+h u^\boldsymbol{\gamma}(h) = \mathbf{x} + h\,\hat{\mathbf{u}}, we have γ′(h)=u^\boldsymbol{\gamma}'(h) = \hat{\mathbf{u}}, so:

g′(h)=∑i=1n∂f∂xi(γ(h)) dγidh=∑i=1n∂f∂xi u^i=∇f(γ(h))⋅u^g'(h) = \sum_{i=1}^{n} \frac{\partial f}{\partial x_i}\big(\boldsymbol{\gamma}(h)\big)\,\frac{d\gamma_i}{dh} = \sum_{i=1}^{n} \frac{\partial f}{\partial x_i}\,\hat{u}_i = \nabla f\big(\boldsymbol{\gamma}(h)\big) \cdot \hat{\mathbf{u}}

Evaluating at h=0h = 0 gives the central result:

  Du^f(x)=∇f(x)⋅u^  \boxed{\;D_{\hat{\mathbf{u}}} f(\mathbf{x}) = \nabla f(\mathbf{x}) \cdot \hat{\mathbf{u}}\;}

The directional derivative in any direction is just the projection of the gradient onto that direction. The gradient packages every directional derivative at once — compute it once, dot it with any u^\hat{\mathbf{u}} you like.

Why the gradient points toward steepest ascent

Expand the dot product geometrically. With θ\theta the angle between ∇f\nabla f and u^\hat{\mathbf{u}}, and ∥u^∥=1\|\hat{\mathbf{u}}\| = 1:

Du^f=∇f⋅u^=∥∇f∥ ∥u^∥cos⁡θ=∥∇f∥cos⁡θD_{\hat{\mathbf{u}}} f = \nabla f \cdot \hat{\mathbf{u}} = \|\nabla f\|\,\|\hat{\mathbf{u}}\|\cos\theta = \|\nabla f\|\cos\theta

Since cos⁡θ\cos\theta ranges over [−1,1][-1, 1], the directional derivative is bounded:

−∥∇f∥  ≤  Du^f  ≤  ∥∇f∥-\|\nabla f\| \;\le\; D_{\hat{\mathbf{u}}} f \;\le\; \|\nabla f\|
Direction u^\hat{\mathbf{u}}θ\thetaDu^fD_{\hat{\mathbf{u}}} fInterpretation
Along ∇f\nabla f00+∥∇f∥+\|\nabla f\|steepest ascent
Against ∇f\nabla fπ\pi−∥∇f∥-\|\nabla f\|steepest descent
Perpendicular to ∇f\nabla fπ/2\pi/200along a level set (contour): ff unchanged

The maximum is attained when θ=0\theta = 0, i.e. u^=∇f/∥∇f∥\hat{\mathbf{u}} = \nabla f / \|\nabla f\|. This proves that the gradient points in the direction of fastest increase, and its magnitude ∥∇f∥\|\nabla f\| is that fastest rate. Gradient descent takes the opposite direction precisely because θ=π\theta = \pi minimizes Du^fD_{\hat{\mathbf{u}}} f.

Gradients are orthogonal to level sets

The third row explains why ∇f\nabla f is always perpendicular to the contour through x\mathbf{x}: moving along a contour leaves ff unchanged, so the directional derivative there is 00, which forces cos⁡θ=0\cos\theta = 0.

A concrete numerical example

Let f(x,y)=x2+3xyf(x, y) = x^2 + 3xy at the point x=(1,2)\mathbf{x} = (1, 2).

Gradient:

∇f=[2x+3y3x]  ⟹  ∇f(1,2)=[2+63]=[83]\nabla f = \begin{bmatrix} 2x + 3y \\ 3x \end{bmatrix} \;\Longrightarrow\; \nabla f(1, 2) = \begin{bmatrix} 2 + 6 \\ 3 \end{bmatrix} = \begin{bmatrix} 8 \\ 3 \end{bmatrix}

Directional derivative toward v=(3,4)\mathbf{v} = (3, 4). First normalize. This is the step beginners skip:

∥v∥=32+42=5,u^=15(3,4)\|\mathbf{v}\| = \sqrt{3^2 + 4^2} = 5, \qquad \hat{\mathbf{u}} = \tfrac{1}{5}(3, 4) Du^f(1,2)=∇f⋅u^=[83]⋅15[34]=8(3)+3(4)5=365=7.2D_{\hat{\mathbf{u}}} f(1,2) = \nabla f \cdot \hat{\mathbf{u}} = \begin{bmatrix} 8 \\ 3 \end{bmatrix} \cdot \tfrac{1}{5}\begin{bmatrix} 3 \\ 4 \end{bmatrix} = \frac{8(3) + 3(4)}{5} = \frac{36}{5} = 7.2

Sanity check against the extremes. The steepest possible rate here is ∥∇f(1,2)∥=82+32=73≈8.54\|\nabla f(1,2)\| = \sqrt{8^2 + 3^2} = \sqrt{73} \approx 8.54. Our value 7.2<8.547.2 < 8.54, as it must be — (3,4)(3,4) is close to but not exactly aligned with (8,3)(8, 3).

Python
import numpy as np

def f(x, y):
    return x**2 + 3*x*y

def grad_f(x, y):
    return np.array([2*x + 3*y, 3*x])

p = (1.0, 2.0)
g = grad_f(*p)                      # [8., 3.]

v = np.array([3.0, 4.0])
u = v / np.linalg.norm(v)           # unit vector [0.6, 0.8]
Du = g @ u                          # directional derivative

print(f"gradient        ∇f = {g}")
print(f"unit direction  û  = {u}")
print(f"D_û f           = {Du:.3f}")          # 7.200
print(f"steepest rate   = {np.linalg.norm(g):.3f}")  # 8.544 (>= 7.200, as required)

# Finite-difference check of the limit definition
h = 1e-6
fd = (f(*(np.array(p) + h*u)) - f(*p)) / h
print(f"finite-diff     = {fd:.3f}")          # ≈ 7.200

Common pitfalls

  1. Forgetting to normalize v\mathbf{v}. Using a non-unit vector inflates the result by ∥v∥\|\mathbf{v}\|. In the example, skipping normalization gives 8(3)+3(4)=368(3)+3(4)=36 instead of 7.27.2.
  2. Assuming partials existing implies the directional derivative exists in every direction. For non-differentiable functions (e.g. f(x,y)=xyx2+y2f(x,y) = \tfrac{xy}{x^2+y^2} at the origin) the chain-rule shortcut ∇f⋅u^\nabla f \cdot \hat{\mathbf{u}} can fail; you must fall back to the limit definition. The dot-product formula is guaranteed only where ff is differentiable.
  3. Treating Du^fD_{\hat{\mathbf{u}}} f as a vector. It is a scalar — a rate. The direction of fastest change is the vector ∇f\nabla f.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Referenced by
The Directional DerivativeDerivatives and GradientsMultivariable OptimizationChain Rule and Backpropag…