Gradient Descent VariantsOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Gradient Descent Variants

Every neural network you have ever used was trained with gradient descent — or, more accurately, with one of its modern descendants. This whole course is the loop's optimization stage in depth: given a fixed model and objective, how do you actually find good parameters? The core idea is simple: follow the negative gradient downhill. The subtleties in how you follow it gave rise to the optimizer zoo: SGD, Momentum, RMSprop, Adam, and a dozen others — each one is a change to this single slot, nothing else.

This lesson traces the lineage, derives each update rule from scratch, and ends with practical guidance on when to reach for each one.

Deep dive
The ML Project Loop
Includes runnable notebook

The gradient descent family

The fundamental update is always the same:

θ←θ−η⋅∇θL\theta \leftarrow \theta - \eta \cdot \nabla_\theta L

What differs between variants is how ∇θL\nabla_\theta L is estimated and how the step is scaled or smoothed.

Full-batch, stochastic, and mini-batch

With nn training examples, the true loss gradient is the average over the whole dataset. But computing that average requires a forward and backward pass through all nn examples — expensive when nn is in the millions.

VariantExamples per stepGradient qualityPractical cost
Full-batch GDAll nnExact1 update per epoch; memory-heavy
Stochastic (SGD)1Noisynn updates per epoch; each is fast
Mini-batch SGD32–512Low-variance estimaten/Bn/B updates per epoch

Mini-batch is the practical sweet spot for two reasons:

  1. GPU vectorization. A GPU executes a batch of BB examples in nearly the same wall-clock time as a single example, so you get BB gradient samples essentially for free. Below B≈32B \approx 32 the GPU sits idle; above B≈512B \approx 512 memory bandwidth becomes the bottleneck.
  2. Noise is a feature. The stochasticity in mini-batch gradients acts as implicit regularization: it pushes the optimizer out of sharp minima (which generalize poorly) and toward flat minima (which generalize well). The next section takes that claim apart, because it is the one people repeat without being able to defend.

Epoch vs. step

One epoch is one full pass over the training set. With batch size BB and dataset size nn, one epoch consists of ⌈n/B⌉\lceil n/B \rceil steps (parameter updates). Most schedules are measured in epochs; most optimizers tune per step.

Why the noise is a regularizer

"SGD's noise helps generalization" is easy to say and hard to justify. The justification has two independent halves, and you need both.

Half one: the noise lets you leave. A mini-batch gradient is the full gradient plus an error term, and the error shrinks with batch size as

σB=σ1B\sigma_B = \frac{\sigma_1}{\sqrt{B}}

Treat the resulting update as a discretized diffusion and SGD acquires a stationary distribution over parameters, p(θ)∝exp⁡ ⁣(−2L(θ)/ησB2)p(\theta) \propto \exp\!\left(-2L(\theta)/\eta\sigma_B^2\right). The quantity T=ησB2T = \eta\sigma_B^2 plays the role of a temperature, and the expected time to escape a basin whose walls rise ΔL\Delta L above its floor is

τ  ∼  exp⁡ ⁣(2 ΔLη σB2)\tau \;\sim\; \exp\!\left(\frac{2\,\Delta L}{\eta\,\sigma_B^2}\right)

Read what that exponent does. Full-batch gradient descent has σB=0\sigma_B = 0, so τ=∞\tau = \infty: it converges to whatever basin it started in and stays there forever, by construction. Halving the batch doubles σB2\sigma_B^2 and takes an exponential bite out of τ\tau. Small batches are not merely jittery, they are ergodic — they keep sampling the landscape instead of committing to the first basin they fall into.

Half two: the basin you end in matters, and flatness is why. Escaping is only useful if the destination is better, and on the training loss it often isn't. The reason to prefer a flat basin is that the training loss is not the thing you are graded on. Your training set is one finite sample; the test loss is what you'd get from a different one, which puts the valley in a slightly different place. To first order, test loss behaves like the training loss displaced by some δ\delta, and a second-order expansion around a minimum θ⋆\theta^\star gives

Ltest(θ⋆)−Ltrain(θ⋆)  ≈  12 δ⊤∇2L δL_{\text{test}}(\theta^\star) - L_{\text{train}}(\theta^\star) \;\approx\; \tfrac{1}{2}\,\delta^{\top} \nabla^2 L\, \delta

The generalization gap is proportional to the curvature at the solution. Same displacement, same data, same everything: a sharp minimum (large Hessian eigenvalues) converts it into a large loss increase, and a flat one absorbs it. This is why "flat minima generalize better" is a statement about robustness to perturbation, not a mystical property of flatness.

Put the halves together and the mechanism is complete: noise decides which minimum you land in, curvature decides what that costs you on data you have not seen, and batch size is the dial on the first.

What the noise is actually for

batch size
sharp: gap 0.2233L″ = 10.17flat: gap 0.0253L″ = 0.62θ — parameter-0.30train loss (what SGD sees)test loss (displaced)
σ = σ₁/√B3.18
live runs in flat basin0 / 0
phaseexploring (0/2700)
escape rate over 240 seeds47%
final train loss−0.2197
final test loss−0.0955
every batch size, 240 seeds each−0.0195−0.1222best10242566432168421batch size (large ← → small, i.e. more noise to the right)final test lossfraction escaping to the flat basin (0 → 1)

Large batches never leave: at B = 1024 every one of the 240 runs finishes in the sharp basin with the best training loss on the chart (−0.2429) and the worst test loss (−0.0195). The best test loss is at B = 4 (−0.1222). Past that the curve turns back up: noise buys exploration, and beyond a point the run simply never settles.

Two minima. The sharp one on the left has the LOWER training loss (−0.2429 against −0.2134) and is 16× more curved. But the test loss is the training loss displaced slightly — a different sample puts the valley somewhere slightly different — and a displacement barely moves you in a wide basin while it throws you up the wall of a narrow one. Every run starts in the sharp basin. Shrink the batch, and the gradient noise σ = σ₁/√B grows until runs start escaping to the flat basin, which is worse on training loss and much better on test loss.

The landscape above is arranged so the greedy answer is the wrong one. The sharp minimum is genuinely deeper on the training loss (−0.2429-0.2429 against −0.2134-0.2134) and 16× more curved, so any procedure that only wants low training loss should prefer it. Watch what the batch size does:

  • B = 1024 down to 32: not one run of 240 escapes. These runs achieve the best training loss on the chart and the worst test loss on the chart. That inversion is the entire argument for calling the noise a regularizer.
  • B = 8: 47% escape, and mean test loss improves from −0.0195-0.0195 to −0.0955-0.0955.
  • B = 4: 65% escape — the best test loss in the sweep.
  • B = 1: the escape rate stops improving while the training loss keeps degrading. A run that never stops bouncing never settles anywhere.

That last point is the honest limit, and it is why the curve turns back up rather than pointing at "smaller is always better". Noise buys exploration and exploration has a price. It also explains the two facts about large-batch training that otherwise look like folklore: why large-batch runs need explicit regularization and learning-rate warm-up to match small-batch generalization, and why the ratio η/B\eta/B (not η\eta alone) is the quantity to hold fixed when you scale a batch size up, since it is ησB2∝η/B\eta\sigma_B^2 \propto \eta/B that sets the temperature.

Flat minima are not the whole story

The sharpness/generalization link is empirical and has real counterexamples: Dinh et al. (2017) showed that for a ReLU network you can rescale one layer up and the next down, leaving the function (and so the test error) completely unchanged while making the minimum arbitrarily sharp by any Hessian-eigenvalue measure. Sharpness is not reparameterization-invariant; test error is. Treat flatness as a useful intuition and a decent predictor, not a theorem.

SGD with momentum

Plain mini-batch SGD bounces back and forth across the walls of narrow ravines (directions where the loss curvature is large), while advancing slowly along the floor where curvature is small. Momentum fixes this by keeping a running velocity that accumulates past gradients.

The update rule

vt=β⋅vt−1+∇Ltv_t = \beta \cdot v_{t-1} + \nabla L_t θt=θt−1−η⋅vt\theta_t = \theta_{t-1} - \eta \cdot v_t

β∈[0,1)\beta \in [0, 1) is the momentum coefficient (typically 0.90.9). The velocity vtv_t accumulates past gradients with exponential decay. Expanding the recursion:

vt=∑k=0tβk ∇Lt−kv_t = \sum_{k=0}^{t} \beta^k \,\nabla L_{t-k}

The weight on the gradient from kk steps ago decays as βk\beta^k, so recent gradients dominate. The effective look-back window is approximately:

effective window≈11−β\text{effective window} \approx \frac{1}{1-\beta}

With β=0.9\beta = 0.9 this is ≈10\approx 10 steps: the velocity is primarily driven by the last 10 gradients.

Heavy ball vs. EMA momentum

This undamped form (vt=βvt−1+∇Ltv_t = \beta v_{t-1} + \nabla L_t) is the heavy-ball rule — the convention used by PyTorch's SGD(momentum=β). Some textbooks write vt=βvt−1+(1−β)∇Ltv_t = \beta v_{t-1} + (1{-}\beta)\nabla L_t (EMA form), which normalizes the velocity magnitude. Adam's first moment (below) uses the EMA form. The two are equivalent up to a rescaling of η\eta.

Worked example: 2 steps with β=0.9\beta = 0.9

Consider minimizing L(θ)=θ2L(\theta) = \theta^2 (gradient =2θ= 2\theta) starting at θ0=3.0\theta_0 = 3.0, with η=0.1\eta = 0.1 and v0=0v_0 = 0.

Step 1:

g1=2(3.0)=6.0g_1 = 2(3.0) = 6.0 v1=0.9⋅0+6.0=6.0v_1 = 0.9 \cdot 0 + 6.0 = 6.0 θ1=3.0−0.1⋅6.0=2.4\theta_1 = 3.0 - 0.1 \cdot 6.0 = 2.4

Step 2:

g2=2(2.4)=4.8g_2 = 2(2.4) = 4.8 v2=0.9⋅6.0+4.8=5.4+4.8=10.2v_2 = 0.9 \cdot 6.0 + 4.8 = 5.4 + 4.8 = 10.2 θ2=2.4−0.1⋅10.2=1.38\theta_2 = 2.4 - 0.1 \cdot 10.2 = 1.38

Plain SGD (no momentum) gives θ1=2.4\theta_1 = 2.4 (identical on step 1 — no accumulated velocity yet) and θ2=2.4−0.1⋅4.8=1.92\theta_2 = 2.4 - 0.1 \cdot 4.8 = 1.92. By step 2, momentum's accumulated velocity has taken a larger step and sits closer to the minimum (θ2=1.38\theta_2 = 1.38 vs 1.921.92).

Do not over-read those two steps, though. Keep this exact run going and momentum sails past zero at step 4 and needs 51 steps to settle inside ∣θ∣<0.01|\theta| < 0.01 against SGD's 26 — on a 1-D bowl there is no cross-valley oscillation for it to cancel, so all the velocity buys you is overshoot. Momentum earns its keep on a ravine, where the cross-valley velocity components cancel while the consistent downhill component accumulates. The trace below steps through both cases.

Why momentum helps: the Rosenbrock function

The Rosenbrock "banana" function is a classic stress test for optimizers:

f(x,y)=(1−x)2+100 (y−x2)2f(x, y) = (1 - x)^2 + 100\,(y - x^2)^2

Its minimum is at (1,1)(1, 1) but it lies inside a long curved valley. Plain SGD follows the steep walls in tiny zig-zags and barely advances along the valley floor. With β=0.9\beta = 0.9, the cross-valley oscillations cancel in the accumulated velocity while the along-valley component accumulates — momentum slides efficiently toward (1,1)(1, 1).

Ball rolling downhill

Momentum behaves exactly like a ball rolling down a hill: it picks up speed on long consistent slopes and isn't deflected much by small bumps (high-frequency gradient noise).

RMSprop

Momentum attacks one problem (oscillation); RMSprop attacks a different one: parameters that receive sparse updates. In an embedding layer, most gradient entries are zero most of the time — a word that never appears in a batch gets zero gradient. Using a single global learning rate means the rare-but-important features get the same (often too-small) step as the dense ones.

Per-parameter adaptive learning rates

RMSprop maintains a per-parameter exponential moving average of squared gradients:

Gt=ρ⋅Gt−1+(1−ρ)⋅(∇Lt)2G_t = \rho \cdot G_{t-1} + (1-\rho) \cdot (\nabla L_t)^2

The update then divides by Gt+ε\sqrt{G_t + \varepsilon}:

θt=θt−1−ηGt+ε⋅∇Lt\theta_t = \theta_{t-1} - \frac{\eta}{\sqrt{G_t + \varepsilon}} \cdot \nabla L_t

Here ρ≈0.9\rho \approx 0.9 and ε≈10−8\varepsilon \approx 10^{-8} (prevents division by zero). The effect is:

  • Large GtG_t (parameter updated frequently with large gradients): effective step ≈η/Gt\approx \eta / \sqrt{G_t} is small — the optimizer is cautious about parameters that already move a lot.
  • Small GtG_t (parameter updated rarely or with small gradients): effective step is large — the optimizer takes a bigger step for sparse features.

This is exactly what you want for sparse inputs: infrequent updates arrive with larger effective learning rates and can catch up.

RMSprop vs. AdaGrad

AdaGrad accumulates squared gradients without discounting, so the effective learning rate shrinks to near zero over long training runs. RMSprop replaces the cumulative sum with an EMA (controlled by ρ\rho), keeping the effective learning rate from decaying to zero.

Adam

Adam (Adaptive Moment Estimation) combines momentum's first moment with RMSprop's second moment, and adds bias correction to fix the underestimation at early timesteps.

The full update

First moment (momentum):

mt=β1⋅mt−1+(1−β1)⋅∇Ltm_t = \beta_1 \cdot m_{t-1} + (1-\beta_1) \cdot \nabla L_t

Second moment (squared gradient EMA):

vt=β2⋅vt−1+(1−β2)⋅(∇Lt)2v_t = \beta_2 \cdot v_{t-1} + (1-\beta_2) \cdot (\nabla L_t)^2

Bias-corrected estimates:

m^t=mt1−β1t,v^t=vt1−β2t\hat{m}_t = \frac{m_t}{1 - \beta_1^t}, \qquad \hat{v}_t = \frac{v_t}{1 - \beta_2^t}

Parameter update:

θt=θt−1−ηv^t+ε⋅m^t\theta_t = \theta_{t-1} - \frac{\eta}{\sqrt{\hat{v}_t} + \varepsilon} \cdot \hat{m}_t

Default hyperparameters: β1=0.9\beta_1 = 0.9, β2=0.999\beta_2 = 0.999, ε=10−8\varepsilon = 10^{-8}.

Why bias correction matters

Both mm and vv are initialized to zero. At t=1t = 1 with β1=0.9\beta_1 = 0.9:

m1=0.9⋅0+0.1⋅g=0.1 gm_1 = 0.9 \cdot 0 + 0.1 \cdot g = 0.1\,g

Without correction, m1m_1 is 10× smaller than the true gradient. It is systematically biased toward zero. Dividing by (1−β11)=0.1(1 - \beta_1^1) = 0.1 recovers gg exactly. The bias decays exponentially: by t=100t = 100, β1100≈3×10−5\beta_1^{100} \approx 3 \times 10^{-5}, so the correction becomes negligible.

Worked example: 2 Adam steps on L(θ)=θ2L(\theta) = \theta^2

Start at θ0=1.0\theta_0 = 1.0, m0=v0=0m_0 = v_0 = 0, η=0.001\eta = 0.001, defaults β1=0.9\beta_1 = 0.9, β2=0.999\beta_2 = 0.999, ε=10−8\varepsilon = 10^{-8}.

Step 1 (t=1t = 1, g1=2×1.0=2.0g_1 = 2 \times 1.0 = 2.0):

m1=0.9(0)+0.1(2.0)=0.2,v1=0.999(0)+0.001(4.0)=0.004m_1 = 0.9(0) + 0.1(2.0) = 0.2, \quad v_1 = 0.999(0) + 0.001(4.0) = 0.004 m^1=0.21−0.9=2.0,v^1=0.0041−0.999=4.0\hat{m}_1 = \frac{0.2}{1 - 0.9} = 2.0, \quad \hat{v}_1 = \frac{0.004}{1 - 0.999} = 4.0 θ1=1.0−0.0014.0+10−8⋅2.0=1.0−0.0022.0=1.0−0.001=0.999\theta_1 = 1.0 - \frac{0.001}{\sqrt{4.0} + 10^{-8}} \cdot 2.0 = 1.0 - \frac{0.002}{2.0} = 1.0 - 0.001 = 0.999

Step 2 (t=2t = 2, g2=2×0.999=1.998g_2 = 2 \times 0.999 = 1.998):

m2=0.9(0.2)+0.1(1.998)=0.18+0.1998=0.3798m_2 = 0.9(0.2) + 0.1(1.998) = 0.18 + 0.1998 = 0.3798 v2=0.999(0.004)+0.001(1.9982)=0.003996+0.003992=0.007988v_2 = 0.999(0.004) + 0.001(1.998^2) = 0.003996 + 0.003992 = 0.007988 m^2=0.37981−0.81=0.37980.19≈1.999,v^2=0.0079881−0.998001≈3.998\hat{m}_2 = \frac{0.3798}{1 - 0.81} = \frac{0.3798}{0.19} \approx 1.999, \quad \hat{v}_2 = \frac{0.007988}{1 - 0.998001} \approx 3.998 θ2=0.999−0.0013.998+10−8⋅1.999≈0.999−0.001=0.998\theta_2 = 0.999 - \frac{0.001}{\sqrt{3.998} + 10^{-8}} \cdot 1.999 \approx 0.999 - 0.001 = 0.998

Notice that m^\hat{m} and v^\hat{v} stay close to their "true" values (gradient ≈2\approx 2, squared gradient ≈4\approx 4) even at t=2t = 2 — that is the bias correction doing its job.

The clearest way to see the difference between these three is to drop them onto the same ill-conditioned bowl. SGD must creep along the steep axis or it zig-zags off the walls; momentum cancels the cross-valley oscillation; Adam rescales each axis so both descend at once. Press play and raise the condition number κ to widen the gap.

Optimizers on a ravine: SGD vs Momentum vs Adam

start
SGD loss0.000
Momentum loss0.002
Adam loss0.000

The loss is an ill-conditioned bowl ½(x² + κy²): steep across the valley, shallow along it. SGD must creep or it zig-zags off the steep walls (rose). Momentum (yellow) cancels the cross-valley oscillation by accumulating velocity. Adam (teal) rescales each axis by its own gradient history, so both directions descend at a healthy rate. Raise κ to sharpen the ravine and widen the gap.

Python
import numpy as np

def adam_step(theta, grad, m, v, t,
              lr=0.001, b1=0.9, b2=0.999, eps=1e-8):
    """One Adam update. Returns (new_theta, new_m, new_v)."""
    m = b1 * m + (1 - b1) * grad         # first moment EMA
    v = b2 * v + (1 - b2) * grad**2      # second moment EMA
    m_hat = m / (1 - b1**t)              # bias correction
    v_hat = v / (1 - b2**t)
    theta = theta - lr * m_hat / (np.sqrt(v_hat) + eps)
    return theta, m, v

# Run 5 steps on L(θ) = θ², grad = 2θ
theta, m, v = 1.0, 0.0, 0.0
for t in range(1, 6):
    g = 2 * theta
    theta, m, v = adam_step(theta, g, m, v, t)
    m_hat = m / (1 - 0.9**t)
    v_hat = v / (1 - 0.999**t)
    print(f"t={t}: θ={theta:.6f}  m={m:.6f}  v={v:.8f}  m̂={m_hat:.4f}  v̂={v_hat:.4f}")

Watch them run

The equations above tell you what each optimizer computes. What they do not show is what each one carries between steps, and how differently that state behaves on an easy problem versus a hard one. Step through all four update rules — first on the L(θ)=θ2L(\theta) = \theta^2 bowl worked above, then on the Rosenbrock valley:

SGD, momentum and Adam side by side — and what each one remembers

1/12

Four optimizers, one loop. Every one of them computes the same gradient at the same point — the only thing that separates them is what they *remember* between steps. SGD remembers nothing. Momentum carries one velocity vector. RMSprop carries a per-parameter average of squared gradients. Adam carries both, plus the step counter t it needs for bias correction. Watch that state column, not the parameter.

python
1def rt(x):
2 return sqrt(x) + 1e-8
3
4def sgd(p, g, lr):
5 return p - lr * g
6
7def momentum(p, g, st, lr, beta=0.9):
8 # heavy ball (PyTorch convention)
9 st.v = beta * st.v + g
10 return p - lr * st.v
11
12def rmsprop(p, g, st, lr, rho=0.9):
13 st.G = rho * st.G + (1-rho) * g**2
14 return p - lr * g / rt(st.G)
15
16def adam(p, g, st, t, lr,
17 b1=0.9, b2=0.999):
18 st.m = b1 * st.m + (1-b1) * g
19 st.v = b2 * st.v + (1-b2) * g**2
20 # both start at 0, so both start
21 # biased low - undo the bias
22 m_hat = st.m / (1 - b1**t)
23 v_hat = st.v / (1 - b2**t)
24 return p - lr * m_hat / rt(v_hat)
state
what each one carries
optimizerstate per parameterwhy
SGD—follow the gradient, nothing else
momentumvaverage away oscillation
RMSpropGrescale by recent gradient size
Adamm, v, tboth, with the t=0 bias undone

Act I runs the first two on L(θ) = θ² from θ₀ = 3.0 with η = 0.1 — the worked example above. Act II moves to Rosenbrock, where the answer changes.

The same four update rules stepped over two problems. Act I is the 1-D bowl worked above: momentum is ahead at step 2, exactly as the algebra says, and then sails past the minimum and takes nearly twice as long as plain SGD to settle — a 1-D bowl has no cross-valley oscillation for it to cancel. Act I(b) shows Adam's two moments and what deleting the bias correction actually does (the first step gets 3× bigger, not smaller). Act II moves to Rosenbrock, where the ordering flips and stays flipped, and ends by measuring the thing that actually distinguishes Adam in practice: not its best-case step count, but how wide a range of learning rates works at all.

What to notice:

  • Momentum is identical to SGD on step 1, and it has to be — the velocity starts at zero, so there is nothing accumulated to differ by.
  • On the 1-D bowl momentum loses. It crosses the minimum at step 4 and takes nearly twice as many steps as SGD to settle. The worked example above stops at step 2, where momentum happens to be ahead.
  • Deleting Adam's bias correction makes the first step 3× bigger, not smaller. Both moments start biased low, but v\sqrt{v} is biased ~31× and mm only 10×, and it is their ratio that reaches the update.
  • On Rosenbrock the ordering flips. SGD spends 88% of its motion bouncing between the walls; momentum's cross-valley components cancel and it covers 7× more ground.
  • Adam's advantage is not speed, it is the width of the target. Momentum's best run here beats Adam's, but Adam converges over ~4 decades of learning rate against SGD's 0.7.

For the full family tree (AdaGrad, AdamW's decoupled weight decay, Nadam), with every update rule derived and compared on the same problem:

Deep dive
Gradient Descent Optimizers: SGD, Momentum, Adagrad, RMSprop & Adam
Includes runnable notebook

Adam does not always generalize best

Adam converges faster than SGD in most settings, but on image classification benchmarks, well-tuned SGD + momentum often achieves 1–2% lower test error. The hypothesis: Adam's adaptive step sizes let it memorize training data more effectively, including noise. When generalization is the top priority and you have compute budget to tune, try both.

Learning rate schedules

A fixed learning rate is rarely optimal. Too large at the end causes the optimizer to bounce around a minimum without settling; too small throughout slows early progress. Schedules vary η\eta over the course of training.

Step decay

Halve (or multiply by some factor γ\gamma) the learning rate every kk epochs:

ηt=η0⋅γ⌊t/k⌋\eta_t = \eta_0 \cdot \gamma^{\lfloor t / k \rfloor}

Simple and effective for standard vision tasks. The discontinuous drops can cause temporary loss spikes that smooth out quickly.

Cosine annealing

Smoothly decay η\eta from ηmax\eta_\text{max} to ηmin\eta_\text{min} over TT epochs following a half cosine:

ηt=ηmin+12(ηmax−ηmin) ⁣(1+cos⁡ ⁣(πtT))\eta_t = \eta_\text{min} + \frac{1}{2}(\eta_\text{max} - \eta_\text{min})\!\left(1 + \cos\!\left(\frac{\pi t}{T}\right)\right)

At t=0t = 0: ηt=ηmax\eta_t = \eta_\text{max} (cosine = 1). At t=Tt = T: ηt=ηmin\eta_t = \eta_\text{min} (cosine = −1). The cosine shape naturally slows the decay near the minimum, giving the optimizer time to settle.

Warm restarts (SGDR) periodically reset t←0t \leftarrow 0, restarting the cosine from ηmax\eta_\text{max}. Each restart can escape a local minimum and explore a new region of the loss surface. Restart intervals are typically doubled after each cycle.

Warm-up phase

Transformers and other large models require a linear warm-up: start from near-zero learning rate and ramp up to ηmax\eta_\text{max} over the first few hundred to few thousand steps, then apply cosine decay.

ηt=ηmax⋅min⁡ ⁣(ttwarmup,  1)(warm-up)\eta_t = \eta_\text{max} \cdot \min\!\left(\frac{t}{t_\text{warmup}},\; 1\right) \quad\text{(warm-up)}

Why? At initialization, the model's weights are random and its gradient estimates are unreliable. A large initial learning rate can cause catastrophic parameter updates before the gradient signal stabilizes. The warm-up gives the optimizer a few steps to build up accurate first- and second-moment estimates before taking large steps.

For schedule shapes beyond these (one-cycle, inverse square root, and how to pick warm-up length), see the deep dive:

Deep dive
Learning Rate Schedules: Warmup, Decay & Restarts
Includes runnable notebook

Adam + cosine is the standard transformer recipe

The original Transformer paper used Adam with β1=0.9\beta_1=0.9, β2=0.98\beta_2=0.98, and a warm-up over 4,0004{,}000 steps followed by decay proportional to t−0.5t^{-0.5}. Most modern LLM recipes use Adam (or AdamW) with linear warm-up followed by cosine annealing to ∼10%\sim 10\% of ηmax\eta_\text{max}.

When to use what

OptimizerBest forCaveat
SGD + momentumComputer vision (ResNets, ConvNets), when generalization mattersRequires more tuning of η\eta and momentum; slow on NLP
AdamNLP / transformers, fast prototyping, any model with sparse gradientsMay generalize slightly worse than SGD on vision
AdamWLarge pre-trained models (GPT, BERT)Adam + decoupled weight decay; prefer over Adam when λ>0\lambda > 0
RMSpropRNNs and LSTMsNon-stationary gradients from backprop-through-time suit the EMA; Adam is usually at least as good
AdagradSparse feature problems (old NLP, recommendation)Effective learning rate shrinks to zero over long runs; replaced by RMSprop / Adam

Practical default

Start with Adam at η=10−3\eta = 10^{-3} and default β1,β2,ε\beta_1, \beta_2, \varepsilon. If training a vision model that you plan to run for a long time and want maximum test accuracy, switch to SGD + momentum 0.9 with a cosine schedule and spend an extra sweep on η\eta.

Exercise

Why does the gradient noise in mini-batch SGD often help generalization compared to full-batch gradient descent?

Need a hint?

Think about the geometry of the loss landscape near flat vs. sharp minima.

Exercise

Why does a flat minimum generalize better than a sharp one of equal training loss? Pick the statement that actually explains the mechanism.

Need a hint?

Write the test loss as the training loss displaced by δ and expand to second order.

Exercise

Modelling SGD as a diffusion gives an expected escape time from a basin of τ ~ exp(2ΔL / ησ_B²), with gradient noise σ_B = σ₁/√B. You double the batch size B and want to keep the exploration behaviour unchanged. What should you do to the learning rate η?

Need a hint?

Find the combination of η and B that appears in the exponent, then hold it fixed.

Exercise

Momentum maintains velocity v_t = β·v_{t-1} + (1-β)·g_t. With β=0.9, how many recent gradient steps does this exponential moving average effectively average over?

Need a hint?

The effective window of an EMA with factor β is approximately 1/(1-β).

Exercise

Adam initializes m_0 = v_0 = 0 and uses bias-corrected estimates m̂_t = m_t / (1-β₁^t). Why is this correction needed at early timesteps?

Need a hint?

What does m_0 = 0 imply about m_1 if β₁ = 0.9 and the true gradient is g?

Common mistakes

  1. Not decaying the learning rate. Keeping a constant η\eta throughout training leaves the optimizer bouncing around the minimum. Add a schedule — even simple step decay helps.
  2. Using Adam's default ε\varepsilon for mixed-precision training. With float16, gradients are small and ε=10−8\varepsilon = 10^{-8} can dominate the denominator. Increase ε\varepsilon to 10−610^{-6} or 10−710^{-7} for mixed-precision.
  3. Forgetting weight decay with Adam. The standard Adam formulation applies weight decay inside the adaptive denominator, which distorts the regularization. Use AdamW (weight_decay as a separate hyperparameter) whenever you want L2 regularization.
  4. Tuning momentum when Adam stalls. Adam stalls are usually a learning rate problem, not a β1\beta_1 problem. Diagnose by plotting the gradient norm: if it explodes, lower η\eta; if it vanishes, raise η\eta or check for dead activations.
  5. Skipping warm-up for transformers. Even a 500-step linear warm-up prevents the early instability that causes NaN losses in large models.

Key takeaways

  • Mini-batch SGD (batch size 32–512) is the practical foundation: it leverages GPU vectorization while keeping gradient noise that improves generalization.
  • Momentum (β≈0.9\beta \approx 0.9) accumulates past gradients with exponential decay (weight βk\beta^k on gradient kk steps back), with an effective look-back window of 1/(1−β)≈101/(1-\beta) \approx 10 steps, smoothing oscillations in narrow ravines.
  • RMSprop normalizes each parameter's update by the EMA of its recent squared gradients, giving large steps to infrequent (sparse) parameters and small steps to frequently-updated ones.
  • Adam combines both: first-moment EMA (momentum) and second-moment EMA (RMSprop), with bias correction that compensates for zero initialization in the first few steps.
  • Learning rate schedules: especially cosine annealing and warm-up — matter as much as the choice of optimizer; a well-scheduled SGD often beats a poorly-scheduled Adam.
  • Default recipe: Adam at η=10−3\eta = 10^{-3} for fast convergence; SGD + momentum + cosine when you need the best generalization on vision tasks.

Related concepts

  • Multivariable Optimization: saddle points, the Hessian, and why the loss surface shape drives optimizer design
  • Gradient Descent: the single-parameter derivation and the effect of learning rate before adding momentum or adaptivity

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelatedGo deeper
Gradient Descent VariantsCalculus for MLNeural NetworksGradient DescentMultivariable OptimizationConvex OptimizationLoss FunctionsHyperparameter Optimizati…Training Techniques & Hyp…The ML Project LoopGradient Descent Optimize…Learning Rate Schedules: …The O(√T) Regret Bound fo…