Multivariable OptimizationOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Multivariable Optimization

Training a neural network is the loop's optimization stage made concrete: find weights w∗\mathbf{w}^* that minimize the loss L(w)\mathcal{L}(\mathbf{w}). In high dimensions, optimization is much richer than the single-variable case. Understanding the geometry of loss landscapes explains why some problems are easy and others are hard.

Real-world examples: Hyperparameter tuning landscapes, neural network loss surfaces, and convex optimization in SVMs are all multivariable optimization problems.

Critical points

A critical point is where the gradient vanishes: ∇f(x∗)=0\nabla f(\mathbf{x}^*) = \mathbf{0}.

Critical points come in three flavors:

TypeDescriptionIn ML
Local minimumLowest in a neighborhoodDesired outcome
Local maximumHighest in a neighborhoodAvoided by gradient descent
Saddle pointLow in some directions, high in othersVery common in deep nets

Worked example: finding the critical points

Take the two-variable function

f(x,y)=x2+xy+y2−3x.f(x, y) = x^2 + xy + y^2 - 3x.

Step 1: compute the gradient. Differentiate with respect to each variable, treating the other as a constant:

∂f∂x=2x+y−3,∂f∂y=x+2y.\frac{\partial f}{\partial x} = 2x + y - 3, \qquad \frac{\partial f}{\partial y} = x + 2y.

So ∇f=[2x+y−3x+2y]\nabla f = \begin{bmatrix} 2x + y - 3 \\ x + 2y \end{bmatrix}.

Step 2: set the gradient to zero. A critical point solves the linear system

{2x+y−3=0x+2y=0\begin{cases} 2x + y - 3 = 0 \\ x + 2y = 0 \end{cases}

From the second equation, x=−2yx = -2y. Substitute into the first:

2(−2y)+y−3=0  ⟹  −4y+y−3=0  ⟹  −3y=3  ⟹  y=−1.2(-2y) + y - 3 = 0 \;\Longrightarrow\; -4y + y - 3 = 0 \;\Longrightarrow\; -3y = 3 \;\Longrightarrow\; y = -1.

Then x=−2(−1)=2x = -2(-1) = 2. The single critical point is x∗=(2,−1)\mathbf{x}^* = (2, -1).

We still do not know what kind of critical point it is — for that we need the Hessian.

Deep networks rarely hit local minima

Research has shown that most critical points in deep network loss landscapes are saddle points, not poor local minima. The loss value at any local minimum in a large network tends to be close to the global minimum. Saddle points are more problematic because gradient descent slows down — the gradient is near zero.

The Hessian matrix

The Hessian H\mathbf{H} is the matrix of second-order partial derivatives:

Hij=∂2f∂xi∂xjH_{ij} = \frac{\partial^2 f}{\partial x_i \partial x_j}

It generalizes the second derivative to multiple dimensions and captures curvature. For a smooth function the mixed partials are equal (∂2f/∂x∂y=∂2f/∂y∂x\partial^2 f / \partial x \partial y = \partial^2 f / \partial y \partial x), so the Hessian is always symmetric, which is why its eigenvalues are real.

Worked example: the Hessian of our function

Continue with f(x,y)=x2+xy+y2−3xf(x, y) = x^2 + xy + y^2 - 3x. Differentiate each first derivative again:

∂2f∂x2=∂∂x(2x+y−3)=2,∂2f∂y2=∂∂y(x+2y)=2,\frac{\partial^2 f}{\partial x^2} = \frac{\partial}{\partial x}(2x + y - 3) = 2, \qquad \frac{\partial^2 f}{\partial y^2} = \frac{\partial}{\partial y}(x + 2y) = 2, ∂2f∂x∂y=∂∂y(2x+y−3)=1,∂2f∂y∂x=∂∂x(x+2y)=1.\frac{\partial^2 f}{\partial x \partial y} = \frac{\partial}{\partial y}(2x + y - 3) = 1, \qquad \frac{\partial^2 f}{\partial y \partial x} = \frac{\partial}{\partial x}(x + 2y) = 1.

Collecting these:

H=[2112].\mathbf{H} = \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix}.

Here the Hessian is constant (the function is quadratic), so it has the same value at every point, including the critical point (2,−1)(2,-1).

Classifying the critical point with eigenvalues

The curvature in each eigendirection is given by the eigenvalues of H\mathbf{H}:

  • Positive definite Hessian (λmin⁡>0\lambda_{\min} > 0): local minimum
  • Negative definite Hessian (λmax⁡<0\lambda_{\max} < 0): local maximum
  • Indefinite Hessian (mixed signs): saddle point

Step 1: solve the characteristic equation det⁡(H−λI)=0\det(\mathbf{H} - \lambda \mathbf{I}) = 0:

det⁡[2−λ112−λ]=(2−λ)2−1=0.\det\begin{bmatrix} 2 - \lambda & 1 \\ 1 & 2 - \lambda \end{bmatrix} = (2 - \lambda)^2 - 1 = 0.

Step 2 — expand and solve:

(2−λ)2=1  ⟹  2−λ=±1  ⟹  λ=1 or λ=3.(2 - \lambda)^2 = 1 \;\Longrightarrow\; 2 - \lambda = \pm 1 \;\Longrightarrow\; \lambda = 1 \ \text{or}\ \lambda = 3.

Both eigenvalues are positive, so H\mathbf{H} is positive definite and (2,−1)(2, -1) is a local minimum (and, since ff is convex, the global minimum). The valley is three times steeper along the λ=3\lambda = 3 eigendirection than along the λ=1\lambda = 1 direction.

Second-order Taylor expansion

Near any point x0\mathbf{x}_0, a smooth function is well approximated by a quadratic:

f(x)≈f(x0)+∇f(x0)⊤(x−x0)+12(x−x0)⊤H(x0)(x−x0)f(\mathbf{x}) \approx f(\mathbf{x}_0) + \nabla f(\mathbf{x}_0)^\top (\mathbf{x} - \mathbf{x}_0) + \frac{1}{2}(\mathbf{x} - \mathbf{x}_0)^\top \mathbf{H}(\mathbf{x}_0)(\mathbf{x} - \mathbf{x}_0)

The three terms are: the value, the linear (slope) correction, and the quadratic (curvature) correction. At a critical point ∇f=0\nabla f = \mathbf{0}, so the linear term drops and the local shape is governed entirely by H\mathbf{H}. This is precisely why the Hessian's eigenvalues decide minimum vs. saddle.

Let s=x−x0\mathbf{s} = \mathbf{x} - \mathbf{x}_0. Writing the curvature term in the eigenbasis H=∑iλivivi⊤\mathbf{H} = \sum_i \lambda_i \mathbf{v}_i \mathbf{v}_i^\top gives

12 s⊤H s=12∑iλi (vi⊤s)2.\tfrac{1}{2}\,\mathbf{s}^\top \mathbf{H}\,\mathbf{s} = \tfrac{1}{2}\sum_i \lambda_i\,(\mathbf{v}_i^\top \mathbf{s})^2.

Each eigendirection contributes 12λi\tfrac{1}{2}\lambda_i times the squared distance moved along it: positive λi\lambda_i curves the bowl up, negative λi\lambda_i curves it down. A mix of signs is exactly a saddle.

Newton's method: one step, derived

Newton's method minimizes the quadratic Taylor model exactly in one step: taking ∇s[∇fTs+12sTHs]=0\nabla_{\mathbf{s}}[\nabla f^T\mathbf{s} + \tfrac{1}{2}\mathbf{s}^T\mathbf{H}\mathbf{s}] = 0 gives s=−H−1∇f\mathbf{s} = -\mathbf{H}^{-1}\nabla f, so the update is x←x−H−1∇f(x)\mathbf{x} \leftarrow \mathbf{x} - \mathbf{H}^{-1}\nabla f(\mathbf{x}). For our ff starting at (0,0)(0,0), this reaches the minimum (2,−1)(2,-1) in exactly one step — compared to dozens of gradient-descent iterations. For the full derivation, the 2×22\times 2 inversion (det⁡H=3\det\mathbf{H}=3), step-by-step numeric verification, and a comparison of Newton vs. GD convergence rates, see:

Deep dive
Newton's Method for Optimization
Includes runnable notebook

Gradient descent on a 2D loss surface

step0
loss11.090

Darker = lower loss. On the convex bowl every path spirals into the single minimum. On the saddle, descent slides away along the y-axis — a critical point where the gradient is zero but it's not a minimum.

Convexity

A function is convex if the line segment between any two points on its graph lies above or on the graph:

f(λx+(1−λ)y)≤λf(x)+(1−λ)f(y),λ∈[0,1]f(\lambda \mathbf{x} + (1-\lambda)\mathbf{y}) \leq \lambda f(\mathbf{x}) + (1-\lambda)f(\mathbf{y}), \quad \lambda \in [0,1]

Equivalently, H⪰0\mathbf{H} \succeq 0 (positive semi-definite) everywhere — every eigenvalue is ≥0\geq 0 at every point.

Why convexity matters:

  • Every local minimum is a global minimum
  • Gradient descent is guaranteed to converge to the global optimum
  • No saddle points

Convex ML problems: Linear regression, logistic regression, SVM, Lasso, Ridge regression. Non-convex ML problems: Neural networks with non-linear activations (any depth ≥ 2).

The eigenvalue convexity test, worked

A quadratic f(x)=12x⊤H xf(\mathbf{x}) = \tfrac{1}{2}\mathbf{x}^\top \mathbf{H}\,\mathbf{x} is convex iff H⪰0\mathbf{H} \succeq 0. Test two matrices by hand using the same 2×22\times2 characteristic equation as before, det⁡(H−λI)=0\det(\mathbf{H} - \lambda\mathbf{I}) = 0.

Convex case H=[2113]\mathbf{H} = \begin{bmatrix} 2 & 1 \\ 1 & 3 \end{bmatrix}:

(2−λ)(3−λ)−1=0  ⟹  λ2−5λ+5=0  ⟹  λ=5±25−202=5±52.(2 - \lambda)(3 - \lambda) - 1 = 0 \;\Longrightarrow\; \lambda^2 - 5\lambda + 5 = 0 \;\Longrightarrow\; \lambda = \frac{5 \pm \sqrt{25 - 20}}{2} = \frac{5 \pm \sqrt{5}}{2}.

That gives λ≈1.38\lambda \approx 1.38 and λ≈3.62\lambda \approx 3.62, both positive — convex (positive definite).

Saddle case H=[1221]\mathbf{H} = \begin{bmatrix} 1 & 2 \\ 2 & 1 \end{bmatrix}:

(1−λ)2−4=0  ⟹  1−λ=±2  ⟹  λ=−1 or λ=3.(1 - \lambda)^2 - 4 = 0 \;\Longrightarrow\; 1 - \lambda = \pm 2 \;\Longrightarrow\; \lambda = -1 \ \text{or}\ \lambda = 3.

One eigenvalue is negative, so H\mathbf{H} is indefinite — not convex, and the origin is a saddle point. Along the eigenvector for λ=3\lambda = 3 the surface curves up; along the eigenvector for λ=−1\lambda = -1 it curves down.

This is exactly the test the companion notebook implements: read off the eigenvalues of H\mathbf{H} and check their signs.

Python
import numpy as np

def classify(H):
    """Classify a critical point from the Hessian's eigenvalues."""
    eigs = np.linalg.eigvalsh(H)        # real eigenvalues (H is symmetric)
    if np.all(eigs > 0):  return "local minimum", eigs
    if np.all(eigs < 0):  return "local maximum", eigs
    return "saddle point", eigs

print(classify(np.array([[2., 1.], [1., 3.]])))  # ('local minimum', [1.38, 3.62])
print(classify(np.array([[1., 2.], [2., 1.]])))  # ('saddle point',  [-1.,  3.])

Learning rate and curvature

The safe gradient descent step size is set by the largest eigenvalue of H\mathbf{H} (the Lipschitz constant of the gradient, LL):

η=1L=1λmax⁡(H)\eta = \frac{1}{L} = \frac{1}{\lambda_{\max}(\mathbf{H})}
  • Too large η\eta: overshoots; oscillates or diverges if η>2/L\eta > 2/L
  • Too small η\eta: slow convergence, especially in flat directions

Worked example: the divergence threshold

Take the separable quadratic f(w)=2w12+12w22f(\mathbf{w}) = 2w_1^2 + \tfrac{1}{2}w_2^2, whose Hessian is

H=[4001],λmax⁡=4,  λmin⁡=1.\mathbf{H} = \begin{bmatrix} 4 & 0 \\ 0 & 1 \end{bmatrix}, \qquad \lambda_{\max} = 4,\ \ \lambda_{\min} = 1.

Gradient descent on a quadratic updates each eigendirection independently. Along the eigendirection with eigenvalue λ\lambda, one step multiplies the coordinate by the factor (1−ηλ)(1 - \eta\lambda):

w←w−η (λw)=(1−ηλ) w.w \leftarrow w - \eta\,(\lambda w) = (1 - \eta\lambda)\,w.

The iteration converges in that direction only when ∣1−ηλ∣<1|1 - \eta\lambda| < 1, i.e. 0<η<2/λ0 < \eta < 2/\lambda. The tightest constraint comes from the largest eigenvalue, so the step diverges as soon as

η>2λmax⁡=24=0.5.\eta > \frac{2}{\lambda_{\max}} = \frac{2}{4} = 0.5.

A standard safe choice is η=1/λmax⁡=0.25\eta = 1/\lambda_{\max} = 0.25, which kills the error along the steepest direction in one step. The rate that optimally balances both directions is η∗=2/(λmax⁡+λmin⁡)=0.4\eta^* = 2/(\lambda_{\max} + \lambda_{\min}) = 0.4 — it contracts each coordinate by the same factor 0.60.6 per step. The notebook runs η∈{0.05,0.2,0.49,0.51}\eta \in \{0.05, 0.2, 0.49, 0.51\} and shows the last value blowing up — exactly the 0.50.5 threshold predicted here.

This is why the curvature of the loss landscape determines the best learning rate, and why adaptive methods like Adam use per-parameter estimates of curvature.

Saddle points and plateaus

In high dimensions, saddle points are exponentially more common than local minima. Gradient descent slows near them because the gradient is small. SGD noise and momentum help escape them by adding randomness to the updates.

Constrained optimization and Lagrange multipliers

To minimize f(x)f(\mathbf{x}) subject to g(x)=0g(\mathbf{x}) = 0, form the Lagrangian:

L(x,λ)=f(x)+λg(x)\mathcal{L}(\mathbf{x}, \lambda) = f(\mathbf{x}) + \lambda g(\mathbf{x})

At the constrained optimum, ∇f=−λ∇g\nabla f = -\lambda \nabla g — the gradient of the objective is parallel to the constraint gradient.

In ML: Weight decay (L2 regularization) is equivalent to constrained optimization: minimize loss subject to ∥w∥2≤r2\|\mathbf{w}\|^2 \leq r^2.

Exercise

At a saddle point, the gradient is zero. How can you identify it (vs a minimum)?

Need a hint?

Look at the Hessian matrix.

Exercise

Which of the following is a convex function?

Need a hint?

Convex functions have a bowl shape — no local minima other than the global one.

Exercise

If the loss landscape has Lipschitz constant L (maximum curvature), what is the optimal gradient descent step size?

Need a hint?

Too large and you overshoot; too small and you crawl.

Common mistakes

  1. Assuming a critical point is a minimum: a gradient of zero could indicate a saddle point, which are far more common in high-dimensional loss surfaces than local minima.
  2. Treating a non-convex loss surface as if it has a unique global minimum: neural network losses are non-convex; there are typically many local minima and saddle points of varying quality.
  3. Choosing a learning rate without considering curvature: second-order methods (Newton, L-BFGS) adapt to curvature; SGD doesn't, which is why the optimal learning rate depends on λmax⁡(H)\lambda_{\max}(\mathbf{H}).

Key takeaways

  • Critical points have ∇f=0\nabla f = 0; the Hessian determines whether they're minima, maxima, or saddle points.
  • Convex losses have guaranteed global optima; neural networks are non-convex.
  • Curvature sets the learning rate: η=1/λmax⁡(H)\eta = 1/\lambda_{\max}(\mathbf{H}) is the safe default, and beyond 2/λmax⁡2/\lambda_{\max} you diverge.
  • Saddle points slow training; SGD noise, momentum, and adaptive methods help escape.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

RelatedGo deeper
Multivariable OptimizationDerivatives and GradientsGradient DescentEigenvalues and Eigenvect…Jacobians & Vector Calcul…Gradient Descent VariantsConvex OptimizationGradient Descent Optimize…The Directional DerivativeNewton's Method for Optim…