Multivariable Optimization
Training a neural network is the loop's optimization stage made concrete: find weights that minimize the loss . In high dimensions, optimization is much richer than the single-variable case. Understanding the geometry of loss landscapes explains why some problems are easy and others are hard.
Real-world examples: Hyperparameter tuning landscapes, neural network loss surfaces, and convex optimization in SVMs are all multivariable optimization problems.
Critical points
A critical point is where the gradient vanishes: .
Critical points come in three flavors:
| Type | Description | In ML |
|---|---|---|
| Local minimum | Lowest in a neighborhood | Desired outcome |
| Local maximum | Highest in a neighborhood | Avoided by gradient descent |
| Saddle point | Low in some directions, high in others | Very common in deep nets |
Worked example: finding the critical points
Take the two-variable function
Step 1: compute the gradient. Differentiate with respect to each variable, treating the other as a constant:
So .
Step 2: set the gradient to zero. A critical point solves the linear system
From the second equation, . Substitute into the first:
Then . The single critical point is .
We still do not know what kind of critical point it is — for that we need the Hessian.
Deep networks rarely hit local minima
Research has shown that most critical points in deep network loss landscapes are saddle points, not poor local minima. The loss value at any local minimum in a large network tends to be close to the global minimum. Saddle points are more problematic because gradient descent slows down — the gradient is near zero.
The Hessian matrix
The Hessian is the matrix of second-order partial derivatives:
It generalizes the second derivative to multiple dimensions and captures curvature. For a smooth function the mixed partials are equal (), so the Hessian is always symmetric, which is why its eigenvalues are real.
Worked example: the Hessian of our function
Continue with . Differentiate each first derivative again:
Collecting these:
Here the Hessian is constant (the function is quadratic), so it has the same value at every point, including the critical point .
Classifying the critical point with eigenvalues
The curvature in each eigendirection is given by the eigenvalues of :
- Positive definite Hessian (): local minimum
- Negative definite Hessian (): local maximum
- Indefinite Hessian (mixed signs): saddle point
Step 1: solve the characteristic equation :
Step 2 — expand and solve:
Both eigenvalues are positive, so is positive definite and is a local minimum (and, since is convex, the global minimum). The valley is three times steeper along the eigendirection than along the direction.
Second-order Taylor expansion
Near any point , a smooth function is well approximated by a quadratic:
The three terms are: the value, the linear (slope) correction, and the quadratic (curvature) correction. At a critical point , so the linear term drops and the local shape is governed entirely by . This is precisely why the Hessian's eigenvalues decide minimum vs. saddle.
Let . Writing the curvature term in the eigenbasis gives
Each eigendirection contributes times the squared distance moved along it: positive curves the bowl up, negative curves it down. A mix of signs is exactly a saddle.
Newton's method: one step, derived
Newton's method minimizes the quadratic Taylor model exactly in one step: taking gives , so the update is . For our starting at , this reaches the minimum in exactly one step — compared to dozens of gradient-descent iterations. For the full derivation, the inversion (), step-by-step numeric verification, and a comparison of Newton vs. GD convergence rates, see:
Gradient descent on a 2D loss surface
Darker = lower loss. On the convex bowl every path spirals into the single minimum. On the saddle, descent slides away along the y-axis — a critical point where the gradient is zero but it's not a minimum.
Convexity
A function is convex if the line segment between any two points on its graph lies above or on the graph:
Equivalently, (positive semi-definite) everywhere — every eigenvalue is at every point.
Why convexity matters:
- Every local minimum is a global minimum
- Gradient descent is guaranteed to converge to the global optimum
- No saddle points
Convex ML problems: Linear regression, logistic regression, SVM, Lasso, Ridge regression. Non-convex ML problems: Neural networks with non-linear activations (any depth ≥ 2).
The eigenvalue convexity test, worked
A quadratic is convex iff . Test two matrices by hand using the same characteristic equation as before, .
Convex case :
That gives and , both positive — convex (positive definite).
Saddle case :
One eigenvalue is negative, so is indefinite — not convex, and the origin is a saddle point. Along the eigenvector for the surface curves up; along the eigenvector for it curves down.
This is exactly the test the companion notebook implements: read off the eigenvalues of and check their signs.
import numpy as np
def classify(H):
"""Classify a critical point from the Hessian's eigenvalues."""
eigs = np.linalg.eigvalsh(H) # real eigenvalues (H is symmetric)
if np.all(eigs > 0): return "local minimum", eigs
if np.all(eigs < 0): return "local maximum", eigs
return "saddle point", eigs
print(classify(np.array([[2., 1.], [1., 3.]]))) # ('local minimum', [1.38, 3.62])
print(classify(np.array([[1., 2.], [2., 1.]]))) # ('saddle point', [-1., 3.])
Learning rate and curvature
The safe gradient descent step size is set by the largest eigenvalue of (the Lipschitz constant of the gradient, ):
- Too large : overshoots; oscillates or diverges if
- Too small : slow convergence, especially in flat directions
Worked example: the divergence threshold
Take the separable quadratic , whose Hessian is
Gradient descent on a quadratic updates each eigendirection independently. Along the eigendirection with eigenvalue , one step multiplies the coordinate by the factor :
The iteration converges in that direction only when , i.e. . The tightest constraint comes from the largest eigenvalue, so the step diverges as soon as
A standard safe choice is , which kills the error along the steepest direction in one step. The rate that optimally balances both directions is — it contracts each coordinate by the same factor per step. The notebook runs and shows the last value blowing up — exactly the threshold predicted here.
This is why the curvature of the loss landscape determines the best learning rate, and why adaptive methods like Adam use per-parameter estimates of curvature.
Saddle points and plateaus
In high dimensions, saddle points are exponentially more common than local minima. Gradient descent slows near them because the gradient is small. SGD noise and momentum help escape them by adding randomness to the updates.
Constrained optimization and Lagrange multipliers
To minimize subject to , form the Lagrangian:
At the constrained optimum, — the gradient of the objective is parallel to the constraint gradient.
In ML: Weight decay (L2 regularization) is equivalent to constrained optimization: minimize loss subject to .
Exercise
At a saddle point, the gradient is zero. How can you identify it (vs a minimum)?
Need a hint?
Look at the Hessian matrix.
Exercise
Which of the following is a convex function?
Need a hint?
Convex functions have a bowl shape — no local minima other than the global one.
Exercise
If the loss landscape has Lipschitz constant L (maximum curvature), what is the optimal gradient descent step size?
Need a hint?
Too large and you overshoot; too small and you crawl.
Common mistakes
- Assuming a critical point is a minimum: a gradient of zero could indicate a saddle point, which are far more common in high-dimensional loss surfaces than local minima.
- Treating a non-convex loss surface as if it has a unique global minimum: neural network losses are non-convex; there are typically many local minima and saddle points of varying quality.
- Choosing a learning rate without considering curvature: second-order methods (Newton, L-BFGS) adapt to curvature; SGD doesn't, which is why the optimal learning rate depends on .
Key takeaways
- Critical points have ; the Hessian determines whether they're minima, maxima, or saddle points.
- Convex losses have guaranteed global optima; neural networks are non-convex.
- Curvature sets the learning rate: is the safe default, and beyond you diverge.
- Saddle points slow training; SGD noise, momentum, and adaptive methods help escape.
Related concepts
- Derivatives and Gradients: the first-order tools
- Gradient Descent: the optimizer that navigates these landscapes
- Eigenvalues and Eigenvectors: eigenvalues of classify critical points
- Gradient Descent Optimizers: momentum, RMSProp, and Adam applied to these landscapes
- The Directional Derivative: why the gradient is the direction of steepest ascent, from the dot-product formula