Gradient Descent Variants
Every neural network you have ever used was trained with gradient descent — or, more accurately, with one of its modern descendants. This whole course is the loop's optimization stage in depth: given a fixed model and objective, how do you actually find good parameters? The core idea is simple: follow the negative gradient downhill. The subtleties in how you follow it gave rise to the optimizer zoo: SGD, Momentum, RMSprop, Adam, and a dozen others — each one is a change to this single slot, nothing else.
This lesson traces the lineage, derives each update rule from scratch, and ends with practical guidance on when to reach for each one.
The gradient descent family
The fundamental update is always the same:
What differs between variants is how is estimated and how the step is scaled or smoothed.
Full-batch, stochastic, and mini-batch
With training examples, the true loss gradient is the average over the whole dataset. But computing that average requires a forward and backward pass through all examples — expensive when is in the millions.
| Variant | Examples per step | Gradient quality | Practical cost |
|---|---|---|---|
| Full-batch GD | All | Exact | 1 update per epoch; memory-heavy |
| Stochastic (SGD) | 1 | Noisy | updates per epoch; each is fast |
| Mini-batch SGD | 32–512 | Low-variance estimate | updates per epoch |
Mini-batch is the practical sweet spot for two reasons:
- GPU vectorization. A GPU executes a batch of examples in nearly the same wall-clock time as a single example, so you get gradient samples essentially for free. Below the GPU sits idle; above memory bandwidth becomes the bottleneck.
- Noise is a feature. The stochasticity in mini-batch gradients acts as implicit regularization: it pushes the optimizer out of sharp minima (which generalize poorly) and toward flat minima (which generalize well). The next section takes that claim apart, because it is the one people repeat without being able to defend.
Epoch vs. step
One epoch is one full pass over the training set. With batch size and dataset size , one epoch consists of steps (parameter updates). Most schedules are measured in epochs; most optimizers tune per step.
Why the noise is a regularizer
"SGD's noise helps generalization" is easy to say and hard to justify. The justification has two independent halves, and you need both.
Half one: the noise lets you leave. A mini-batch gradient is the full gradient plus an error term, and the error shrinks with batch size as
Treat the resulting update as a discretized diffusion and SGD acquires a stationary distribution over parameters, . The quantity plays the role of a temperature, and the expected time to escape a basin whose walls rise above its floor is
Read what that exponent does. Full-batch gradient descent has , so : it converges to whatever basin it started in and stays there forever, by construction. Halving the batch doubles and takes an exponential bite out of . Small batches are not merely jittery, they are ergodic — they keep sampling the landscape instead of committing to the first basin they fall into.
Half two: the basin you end in matters, and flatness is why. Escaping is only useful if the destination is better, and on the training loss it often isn't. The reason to prefer a flat basin is that the training loss is not the thing you are graded on. Your training set is one finite sample; the test loss is what you'd get from a different one, which puts the valley in a slightly different place. To first order, test loss behaves like the training loss displaced by some , and a second-order expansion around a minimum gives
The generalization gap is proportional to the curvature at the solution. Same displacement, same data, same everything: a sharp minimum (large Hessian eigenvalues) converts it into a large loss increase, and a flat one absorbs it. This is why "flat minima generalize better" is a statement about robustness to perturbation, not a mystical property of flatness.
Put the halves together and the mechanism is complete: noise decides which minimum you land in, curvature decides what that costs you on data you have not seen, and batch size is the dial on the first.
What the noise is actually for
Large batches never leave: at B = 1024 every one of the 240 runs finishes in the sharp basin with the best training loss on the chart (−0.2429) and the worst test loss (−0.0195). The best test loss is at B = 4 (−0.1222). Past that the curve turns back up: noise buys exploration, and beyond a point the run simply never settles.
Two minima. The sharp one on the left has the LOWER training loss (−0.2429 against −0.2134) and is 16× more curved. But the test loss is the training loss displaced slightly — a different sample puts the valley somewhere slightly different — and a displacement barely moves you in a wide basin while it throws you up the wall of a narrow one. Every run starts in the sharp basin. Shrink the batch, and the gradient noise σ = σ₁/√B grows until runs start escaping to the flat basin, which is worse on training loss and much better on test loss.
The landscape above is arranged so the greedy answer is the wrong one. The sharp minimum is genuinely deeper on the training loss ( against ) and 16× more curved, so any procedure that only wants low training loss should prefer it. Watch what the batch size does:
- B = 1024 down to 32: not one run of 240 escapes. These runs achieve the best training loss on the chart and the worst test loss on the chart. That inversion is the entire argument for calling the noise a regularizer.
- B = 8: 47% escape, and mean test loss improves from to .
- B = 4: 65% escape — the best test loss in the sweep.
- B = 1: the escape rate stops improving while the training loss keeps degrading. A run that never stops bouncing never settles anywhere.
That last point is the honest limit, and it is why the curve turns back up rather than pointing at "smaller is always better". Noise buys exploration and exploration has a price. It also explains the two facts about large-batch training that otherwise look like folklore: why large-batch runs need explicit regularization and learning-rate warm-up to match small-batch generalization, and why the ratio (not alone) is the quantity to hold fixed when you scale a batch size up, since it is that sets the temperature.
Flat minima are not the whole story
The sharpness/generalization link is empirical and has real counterexamples: Dinh et al. (2017) showed that for a ReLU network you can rescale one layer up and the next down, leaving the function (and so the test error) completely unchanged while making the minimum arbitrarily sharp by any Hessian-eigenvalue measure. Sharpness is not reparameterization-invariant; test error is. Treat flatness as a useful intuition and a decent predictor, not a theorem.
SGD with momentum
Plain mini-batch SGD bounces back and forth across the walls of narrow ravines (directions where the loss curvature is large), while advancing slowly along the floor where curvature is small. Momentum fixes this by keeping a running velocity that accumulates past gradients.
The update rule
is the momentum coefficient (typically ). The velocity accumulates past gradients with exponential decay. Expanding the recursion:
The weight on the gradient from steps ago decays as , so recent gradients dominate. The effective look-back window is approximately:
With this is steps: the velocity is primarily driven by the last 10 gradients.
Heavy ball vs. EMA momentum
This undamped form () is the heavy-ball rule — the convention used by PyTorch's SGD(momentum=β). Some textbooks write (EMA form), which normalizes the velocity magnitude. Adam's first moment (below) uses the EMA form. The two are equivalent up to a rescaling of .
Worked example: 2 steps with
Consider minimizing (gradient ) starting at , with and .
Step 1:
Step 2:
Plain SGD (no momentum) gives (identical on step 1 — no accumulated velocity yet) and . By step 2, momentum's accumulated velocity has taken a larger step and sits closer to the minimum ( vs ).
Do not over-read those two steps, though. Keep this exact run going and momentum sails past zero at step 4 and needs 51 steps to settle inside against SGD's 26 — on a 1-D bowl there is no cross-valley oscillation for it to cancel, so all the velocity buys you is overshoot. Momentum earns its keep on a ravine, where the cross-valley velocity components cancel while the consistent downhill component accumulates. The trace below steps through both cases.
Why momentum helps: the Rosenbrock function
The Rosenbrock "banana" function is a classic stress test for optimizers:
Its minimum is at but it lies inside a long curved valley. Plain SGD follows the steep walls in tiny zig-zags and barely advances along the valley floor. With , the cross-valley oscillations cancel in the accumulated velocity while the along-valley component accumulates — momentum slides efficiently toward .
Ball rolling downhill
Momentum behaves exactly like a ball rolling down a hill: it picks up speed on long consistent slopes and isn't deflected much by small bumps (high-frequency gradient noise).
RMSprop
Momentum attacks one problem (oscillation); RMSprop attacks a different one: parameters that receive sparse updates. In an embedding layer, most gradient entries are zero most of the time — a word that never appears in a batch gets zero gradient. Using a single global learning rate means the rare-but-important features get the same (often too-small) step as the dense ones.
Per-parameter adaptive learning rates
RMSprop maintains a per-parameter exponential moving average of squared gradients:
The update then divides by :
Here and (prevents division by zero). The effect is:
- Large (parameter updated frequently with large gradients): effective step is small — the optimizer is cautious about parameters that already move a lot.
- Small (parameter updated rarely or with small gradients): effective step is large — the optimizer takes a bigger step for sparse features.
This is exactly what you want for sparse inputs: infrequent updates arrive with larger effective learning rates and can catch up.
RMSprop vs. AdaGrad
AdaGrad accumulates squared gradients without discounting, so the effective learning rate shrinks to near zero over long training runs. RMSprop replaces the cumulative sum with an EMA (controlled by ), keeping the effective learning rate from decaying to zero.
Adam
Adam (Adaptive Moment Estimation) combines momentum's first moment with RMSprop's second moment, and adds bias correction to fix the underestimation at early timesteps.
The full update
First moment (momentum):
Second moment (squared gradient EMA):
Bias-corrected estimates:
Parameter update:
Default hyperparameters: , , .
Why bias correction matters
Both and are initialized to zero. At with :
Without correction, is 10× smaller than the true gradient. It is systematically biased toward zero. Dividing by recovers exactly. The bias decays exponentially: by , , so the correction becomes negligible.
Worked example: 2 Adam steps on
Start at , , , defaults , , .
Step 1 (, ):
Step 2 (, ):
Notice that and stay close to their "true" values (gradient , squared gradient ) even at — that is the bias correction doing its job.
The clearest way to see the difference between these three is to drop them onto the same ill-conditioned bowl. SGD must creep along the steep axis or it zig-zags off the walls; momentum cancels the cross-valley oscillation; Adam rescales each axis so both descend at once. Press play and raise the condition number κ to widen the gap.
Optimizers on a ravine: SGD vs Momentum vs Adam
The loss is an ill-conditioned bowl ½(x² + κy²): steep across the valley, shallow along it. SGD must creep or it zig-zags off the steep walls (rose). Momentum (yellow) cancels the cross-valley oscillation by accumulating velocity. Adam (teal) rescales each axis by its own gradient history, so both directions descend at a healthy rate. Raise κ to sharpen the ravine and widen the gap.
import numpy as np
def adam_step(theta, grad, m, v, t,
lr=0.001, b1=0.9, b2=0.999, eps=1e-8):
"""One Adam update. Returns (new_theta, new_m, new_v)."""
m = b1 * m + (1 - b1) * grad # first moment EMA
v = b2 * v + (1 - b2) * grad**2 # second moment EMA
m_hat = m / (1 - b1**t) # bias correction
v_hat = v / (1 - b2**t)
theta = theta - lr * m_hat / (np.sqrt(v_hat) + eps)
return theta, m, v
# Run 5 steps on L(θ) = θ², grad = 2θ
theta, m, v = 1.0, 0.0, 0.0
for t in range(1, 6):
g = 2 * theta
theta, m, v = adam_step(theta, g, m, v, t)
m_hat = m / (1 - 0.9**t)
v_hat = v / (1 - 0.999**t)
print(f"t={t}: θ={theta:.6f} m={m:.6f} v={v:.8f} m̂={m_hat:.4f} v̂={v_hat:.4f}")
Watch them run
The equations above tell you what each optimizer computes. What they do not show is what each one carries between steps, and how differently that state behaves on an easy problem versus a hard one. Step through all four update rules — first on the bowl worked above, then on the Rosenbrock valley:
SGD, momentum and Adam side by side — and what each one remembers
Four optimizers, one loop. Every one of them computes the same gradient at the same point — the only thing that separates them is what they *remember* between steps. SGD remembers nothing. Momentum carries one velocity vector. RMSprop carries a per-parameter average of squared gradients. Adam carries both, plus the step counter t it needs for bias correction. Watch that state column, not the parameter.
| optimizer | state per parameter | why |
|---|---|---|
| SGD | — | follow the gradient, nothing else |
| momentum | v | average away oscillation |
| RMSprop | G | rescale by recent gradient size |
| Adam | m, v, t | both, with the t=0 bias undone |
Act I runs the first two on L(θ) = θ² from θ₀ = 3.0 with η = 0.1 — the worked example above. Act II moves to Rosenbrock, where the answer changes.
The same four update rules stepped over two problems. Act I is the 1-D bowl worked above: momentum is ahead at step 2, exactly as the algebra says, and then sails past the minimum and takes nearly twice as long as plain SGD to settle — a 1-D bowl has no cross-valley oscillation for it to cancel. Act I(b) shows Adam's two moments and what deleting the bias correction actually does (the first step gets 3× bigger, not smaller). Act II moves to Rosenbrock, where the ordering flips and stays flipped, and ends by measuring the thing that actually distinguishes Adam in practice: not its best-case step count, but how wide a range of learning rates works at all.
What to notice:
- Momentum is identical to SGD on step 1, and it has to be — the velocity starts at zero, so there is nothing accumulated to differ by.
- On the 1-D bowl momentum loses. It crosses the minimum at step 4 and takes nearly twice as many steps as SGD to settle. The worked example above stops at step 2, where momentum happens to be ahead.
- Deleting Adam's bias correction makes the first step 3× bigger, not smaller. Both moments start biased low, but is biased ~31× and only 10×, and it is their ratio that reaches the update.
- On Rosenbrock the ordering flips. SGD spends 88% of its motion bouncing between the walls; momentum's cross-valley components cancel and it covers 7× more ground.
- Adam's advantage is not speed, it is the width of the target. Momentum's best run here beats Adam's, but Adam converges over ~4 decades of learning rate against SGD's 0.7.
For the full family tree (AdaGrad, AdamW's decoupled weight decay, Nadam), with every update rule derived and compared on the same problem:
Adam does not always generalize best
Adam converges faster than SGD in most settings, but on image classification benchmarks, well-tuned SGD + momentum often achieves 1–2% lower test error. The hypothesis: Adam's adaptive step sizes let it memorize training data more effectively, including noise. When generalization is the top priority and you have compute budget to tune, try both.
Learning rate schedules
A fixed learning rate is rarely optimal. Too large at the end causes the optimizer to bounce around a minimum without settling; too small throughout slows early progress. Schedules vary over the course of training.
Step decay
Halve (or multiply by some factor ) the learning rate every epochs:
Simple and effective for standard vision tasks. The discontinuous drops can cause temporary loss spikes that smooth out quickly.
Cosine annealing
Smoothly decay from to over epochs following a half cosine:
At : (cosine = 1). At : (cosine = −1). The cosine shape naturally slows the decay near the minimum, giving the optimizer time to settle.
Warm restarts (SGDR) periodically reset , restarting the cosine from . Each restart can escape a local minimum and explore a new region of the loss surface. Restart intervals are typically doubled after each cycle.
Warm-up phase
Transformers and other large models require a linear warm-up: start from near-zero learning rate and ramp up to over the first few hundred to few thousand steps, then apply cosine decay.
Why? At initialization, the model's weights are random and its gradient estimates are unreliable. A large initial learning rate can cause catastrophic parameter updates before the gradient signal stabilizes. The warm-up gives the optimizer a few steps to build up accurate first- and second-moment estimates before taking large steps.
For schedule shapes beyond these (one-cycle, inverse square root, and how to pick warm-up length), see the deep dive:
Adam + cosine is the standard transformer recipe
The original Transformer paper used Adam with , , and a warm-up over steps followed by decay proportional to . Most modern LLM recipes use Adam (or AdamW) with linear warm-up followed by cosine annealing to of .
When to use what
| Optimizer | Best for | Caveat |
|---|---|---|
| SGD + momentum | Computer vision (ResNets, ConvNets), when generalization matters | Requires more tuning of and momentum; slow on NLP |
| Adam | NLP / transformers, fast prototyping, any model with sparse gradients | May generalize slightly worse than SGD on vision |
| AdamW | Large pre-trained models (GPT, BERT) | Adam + decoupled weight decay; prefer over Adam when |
| RMSprop | RNNs and LSTMs | Non-stationary gradients from backprop-through-time suit the EMA; Adam is usually at least as good |
| Adagrad | Sparse feature problems (old NLP, recommendation) | Effective learning rate shrinks to zero over long runs; replaced by RMSprop / Adam |
Practical default
Start with Adam at and default . If training a vision model that you plan to run for a long time and want maximum test accuracy, switch to SGD + momentum 0.9 with a cosine schedule and spend an extra sweep on .
Exercise
Why does the gradient noise in mini-batch SGD often help generalization compared to full-batch gradient descent?
Need a hint?
Think about the geometry of the loss landscape near flat vs. sharp minima.
Exercise
Why does a flat minimum generalize better than a sharp one of equal training loss? Pick the statement that actually explains the mechanism.
Need a hint?
Write the test loss as the training loss displaced by δ and expand to second order.
Exercise
Modelling SGD as a diffusion gives an expected escape time from a basin of τ ~ exp(2ΔL / ησ_B²), with gradient noise σ_B = σ₁/√B. You double the batch size B and want to keep the exploration behaviour unchanged. What should you do to the learning rate η?
Need a hint?
Find the combination of η and B that appears in the exponent, then hold it fixed.
Exercise
Momentum maintains velocity v_t = β·v_{t-1} + (1-β)·g_t. With β=0.9, how many recent gradient steps does this exponential moving average effectively average over?
Need a hint?
The effective window of an EMA with factor β is approximately 1/(1-β).
Exercise
Adam initializes m_0 = v_0 = 0 and uses bias-corrected estimates m̂_t = m_t / (1-β₁^t). Why is this correction needed at early timesteps?
Need a hint?
What does m_0 = 0 imply about m_1 if β₁ = 0.9 and the true gradient is g?
Common mistakes
- Not decaying the learning rate. Keeping a constant throughout training leaves the optimizer bouncing around the minimum. Add a schedule — even simple step decay helps.
- Using Adam's default for mixed-precision training. With float16, gradients are small and can dominate the denominator. Increase to or for mixed-precision.
- Forgetting weight decay with Adam. The standard Adam formulation applies weight decay inside the adaptive denominator, which distorts the regularization. Use AdamW (
weight_decayas a separate hyperparameter) whenever you want L2 regularization. - Tuning momentum when Adam stalls. Adam stalls are usually a learning rate problem, not a problem. Diagnose by plotting the gradient norm: if it explodes, lower ; if it vanishes, raise or check for dead activations.
- Skipping warm-up for transformers. Even a 500-step linear warm-up prevents the early instability that causes NaN losses in large models.
Key takeaways
- Mini-batch SGD (batch size 32–512) is the practical foundation: it leverages GPU vectorization while keeping gradient noise that improves generalization.
- Momentum () accumulates past gradients with exponential decay (weight on gradient steps back), with an effective look-back window of steps, smoothing oscillations in narrow ravines.
- RMSprop normalizes each parameter's update by the EMA of its recent squared gradients, giving large steps to infrequent (sparse) parameters and small steps to frequently-updated ones.
- Adam combines both: first-moment EMA (momentum) and second-moment EMA (RMSprop), with bias correction that compensates for zero initialization in the first few steps.
- Learning rate schedules: especially cosine annealing and warm-up — matter as much as the choice of optimizer; a well-scheduled SGD often beats a poorly-scheduled Adam.
- Default recipe: Adam at for fast convergence; SGD + momentum + cosine when you need the best generalization on vision tasks.
Related concepts
- Multivariable Optimization: saddle points, the Hessian, and why the loss surface shape drives optimizer design
- Gradient Descent: the single-parameter derivation and the effect of learning rate before adding momentum or adaptivity