Newton's Method for Optimization
Newton's method minimizes a function by fitting a local quadratic model at the current point and jumping directly to its minimum. For a true quadratic this lands at the global minimum in one step. For non-quadratic functions it converges much faster than gradient descent near a minimum.
The core idea behind Newton's method
Near a point , any smooth is approximated by the second-order Taylor expansion:
where is the Hessian (matrix of second derivatives).
Write the displacement as . Newton's method minimizes the right-hand side over , ignoring the constant .
Deriving the Newton step
Take the gradient of the quadratic model with respect to and set it to zero:
Solve for :
The Newton update is:
Compare to gradient descent : Newton replaces the scalar learning rate with the inverse Hessian , which rescales and rotates the gradient to account for curvature.
Curvature correction
In a steep direction ( large), shrinks the step — preventing overshoot. In a flat direction ( small), it enlarges the step — accelerating progress. Gradient descent uses the same step size everywhere, so it struggles when the loss landscape is elongated (ill-conditioned).
Watch it run
The step computed on the function worked below, with the level sets drawn so you can see where the step points:
Newton's method — one step to the minimum, and why we can't always take it
f(x, y) = x² + xy + y² − 3x, a convex quadratic whose minimum sits at (2, −1) with f = -3. Start at the origin. The contours are ellipses rather than circles — the xy term tilts them — and that tilt is exactly what a single learning rate cannot cope with.
Newton's method on f(x, y) = x² + xy + y² − 3x from the origin, the quadratic worked above. Watch the y-component of the step: the gradient's is zero, yet the correct move has y = −1, and the Hessian's off-diagonal term is the only thing that knows it. One step lands exactly on the minimum. The final steps run gradient descent on the same function from the same point for comparison, and then state plainly what Newton costs.
What to notice as you step:
- The gradient's -component is zero at the origin, but the correct step's is . The minimum is at and the gradient has nothing to say about it; the Hessian's off-diagonal term is the only thing that knows and are coupled. This is the clearest possible statement of what curvature buys.
- One step, exactly on the optimum, and the gradient there is exactly zero. For a quadratic the second-order Taylor model is the function, so minimizing the model minimizes the function.
- The comparison step runs gradient descent on the same function from the same point: 22 steps at , divergence at , and no that fixes it — the condition number is only 3 here, and real problems are far worse.
- The final step is the bill: memory, per solve, and steps that move uphill wherever the Hessian is not positive definite.
A single step landing exactly on the minimum ()
The function has a unique minimum at (verified by the Hessian analysis in the multivariable optimization lesson).
Gradient and Hessian at :
(Constant because is quadratic.)
Invert the Hessian. For a matrix:
Newton step from :
Newton lands exactly on the minimum in one step. This is the defining property: for a quadratic, the Taylor model is the function, so minimizing the model is the same as minimizing the function.
Comparison to gradient descent. A plain gradient step from with learning rate :
This only moves along the -axis regardless of — it never escapes to . Gradient descent would require dozens of iterations (zigzagging due to the off-diagonal coupling in ) to converge to .
import numpy as np
def f(x, y):
return x**2 + x*y + y**2 - 3*x
def grad_f(x, y):
return np.array([2*x + y - 3, x + 2*y])
H = np.array([[2., 1.], [1., 2.]])
H_inv = np.linalg.inv(H)
# Newton step from (0, 0)
x0 = np.array([0., 0.])
g = grad_f(*x0)
x1 = x0 - H_inv @ g
print(f"Starting point: {x0} f = {f(*x0):.3f}") # [0., 0.] f = 0.000
print(f"Gradient at x0: {g}") # [-3., 0.]
print(f"Newton step: {x1} f = {f(*x1):.3f}") # [2., -1.] f = -3.000
print(f"Gradient at x1: {grad_f(*x1)}") # [0., 0.] — we're at the minimum
# Gradient descent for comparison: many steps needed
lr = 0.1
xgd = np.array([0., 0.])
for i in range(50):
xgd -= lr * grad_f(*xgd)
print(f"GD after 50 steps: {xgd.round(3)} f = {f(*xgd):.3f}") # ≈ [2., -1.]
Newton vs gradient descent: iteration count and cost
| Method | Steps to reach | Cost per step |
|---|---|---|
| Gradient descent () | ||
| Newton's method | (quadratic) | (matrix inversion) |
The cost to compute and invert is the main limitation: for a neural network with millions of parameters , this is completely infeasible. Practical approximations:
- Quasi-Newton (L-BFGS): builds a low-rank approximation of from gradient history — per step with stored vectors.
- Diagonal approximation (Adam, RMSProp): keeps only the diagonal of , one number per parameter — per step.
Adam as an approximate Newton
Adam maintains per-parameter running estimates of the second moment of the gradient, which approximates the diagonal of the Hessian. Dividing the gradient by is a diagonal Newton step, which is why Adam converges faster than vanilla SGD on ill-conditioned loss surfaces.
Related concepts
- Multivariable Optimization: Hessian, critical point classification, convexity, and learning rate bounds
- Gradient Descent: the simpler first-order method Newton improves upon
- Eigenvalues and Eigenvectors: eigenvalues of determine Newton step directions