Gradient Descent Optimizers: SGD, Momentum, Adagrad, RMSprop & Adam
Every optimizer answers the same question — given the gradient, how do I update the weights? — and they differ in how much memory of the past they keep and whether they adapt the step size per parameter. This page lays out the update rules in one consistent notation so the lineage from plain SGD to Adam is visible as a sequence of small, well-motivated patches.
Throughout, are the parameters, is the gradient at step , and is the learning rate.
SGD: the baseline
Stochastic gradient descent takes one step downhill per mini-batch:
Simple, memory-free, and with a good learning-rate schedule it generalizes excellently — well-tuned SGD with momentum still wins some vision benchmarks. Its weaknesses motivate everything that follows: it crawls through ravines (steep in one direction, flat in another), treats every parameter with the same step size, and is sensitive to the learning rate.
Momentum: remember the direction
Accumulate a running velocity of past gradients so consistent directions build speed and oscillations cancel:
with momentum coefficient . In a ravine, the back-and-forth components cancel across steps while the consistent downhill component accumulates — like a ball gaining speed on a long slope.
Nesterov (NAG) evaluates the gradient at the look-ahead point rather than at . By "peeking" where momentum is about to carry it, it corrects overshoot a step earlier and converges slightly faster.
The adaptive family: a per-parameter step size
The next idea: scale each parameter's step by its own gradient history, so rarely-active features get large steps and frequently-updated ones get small steps.
Adagrad
Accumulate the sum of squared gradients (per parameter) and divide:
Great for sparse features. Fatal flaw: only grows, so the effective rate decays monotonically to zero — training stalls before reaching a good solution.
RMSprop
Fix Adagrad's decay-to-zero by replacing the ever-growing sum with an exponential moving average of squared gradients:
with . The average forgets old gradients, so the effective rate stops shrinking and training continues. Adadelta is a close cousin that also removes the need to set at all by tracking a second moving average of the updates.
Adam: momentum meets adaptivity
Adam combines momentum (first moment) and RMSprop (second moment), each as an exponential moving average:
Because and start at zero, they are biased toward zero early in training. Adam applies bias correction:
Defaults , , work across a huge range of problems with little tuning, which is why Adam is the default for most deep learning.
Why bias correction matters early
At with , the raw second moment is — about 1000× too small, which would make the first step enormous. The correction divides by , restoring the right scale. The factors and both as grows, so bias correction only affects the opening steps — exactly where it is needed.
AdamW: decoupled weight decay
Standard Adam implements L2 regularization by adding to the gradient, but the adaptive denominator then scales the regularization unevenly across parameters. AdamW decouples weight decay from the gradient step, applying it directly to the weights:
This restores proper regularization and is the standard optimizer for training transformers. (Nadam is Adam with Nesterov-style momentum; a smaller, optional tweak.)
Comparison
| Optimizer | State per param | Adapts per-param rate | Notes |
|---|---|---|---|
| SGD | none | ✗ | strong with a schedule; best generalization on some tasks |
| Momentum | velocity | ✗ | escapes ravines; near-free upgrade over SGD |
| Adagrad | sum of | ✓ | good for sparse data; rate decays to zero |
| RMSprop | EMA of | ✓ | fixes Adagrad's stall |
| Adam | EMA of and | ✓ | robust default, fast convergence |
| AdamW | EMA of and | ✓ | Adam + correct weight decay: transformer default |
When each fails
- SGD without momentum stalls in ravines and is acutely learning-rate sensitive.
- Adagrad halts prematurely as its accumulated denominator grows without bound.
- Adam can converge to solutions that generalize slightly worse than SGD on some vision tasks, and rare non-convergence cases motivated fixes like AMSGrad. It also still needs warmup early (unstable second-moment estimates) — see the schedules page.
Adaptive ≠ no tuning
"Adam just works" is mostly true, but the learning rate still matters and the right value differs from SGD's (Adam's effective per-parameter scaling means its lives on a different scale, typically to ). And almost all modern recipes pair the optimizer with a schedule — AdamW + linear-warmup-cosine is the workhorse combination.
Related concepts
- Gradient Descent: the base update and the ravine problem these optimizers solve
- Multivariable Optimization: gradients, curvature, and why per-parameter scaling helps
- Learning Rate Schedules: the schedule that sits on top of any optimizer; warmup is essential for Adam
- Newton's Method: the second-order ideal these first-order methods cheaply approximate