Learning Rate Schedules: Warmup, Decay & RestartsOpen in Colab

Learning Rate Schedules: Warmup, Decay & Restarts

The learning rate η\eta is the single most important training hyperparameter, but the best value changes over the course of training. Early on you want large steps to traverse the loss landscape quickly; near a minimum you want small steps to settle without bouncing. A learning rate schedule varies η\eta across steps to get both, and modern recipes (especially for transformers) treat the schedule as essential, not optional.

Why a constant rate is a compromise

A fixed η\eta forces one value to serve two conflicting phases. Too large for the whole run and training oscillates around the minimum, never settling — the loss plateaus high. Too small and early training crawls, wasting compute and risking getting stuck in a poor region. Decaying the rate over time resolves the tension: big steps while far away, small steps to fine-tune. Empirically, a well-chosen decay schedule reaches a lower final loss than any constant rate.

Decay schedules

Let η0\eta_0 be the base rate and tt the step (or epoch).

Step decay

Drop the rate by a factor γ\gamma every ss steps:

ηt=η0⋅γ⌊t/s⌋\eta_t = \eta_0 \cdot \gamma^{\lfloor t / s \rfloor}

Simple and a long-standing default for vision (e.g. ×0.1 at epochs 30, 60, 90). The discontinuous drops are visible as sudden loss improvements.

Exponential decay

Smooth version of step decay: ηt=η0 e−λt\eta_t = \eta_0 \, e^{-\lambda t}. No cliffs, one decay-rate knob.

Cosine annealing

The modern default. Anneal η\eta along a half-cosine from η0\eta_0 down to a floor \eta_\min over TT steps:

\eta_t = \eta_\min + \tfrac{1}{2}(\eta_0 - \eta_\min)\left(1 + \cos\frac{\pi t}{T}\right)

It spends a long time at high rates early, then decays gently — the shape that pairs well with warmup for transformers and large-batch training.

Polynomial / linear decay

ηt=η0(1−t/T)p\eta_t = \eta_0 (1 - t/T)^p. With p=1p=1 this is the linear decay-to-zero used in many LLM fine-tuning recipes.

Warmup

Starting at the full learning rate can be destabilizing — especially for adaptive optimizers and large batches. Warmup ramps η\eta up from ~0 to η0\eta_0 over the first TwT_w steps (usually linearly), then hands off to a decay schedule:

ηt=η0⋅tTw(t≤Tw),then decay for t>Tw\eta_t = \eta_0 \cdot \frac{t}{T_w} \quad (t \le T_w), \qquad \text{then decay for } t > T_w

Why Adam needs warmup

Adam divides each step by a running estimate of the gradient's second moment. In the first few steps that estimate is computed from almost no data, so it is high-variance — early updates can be wildly mis-scaled and shove the weights into a bad region. Warmup keeps steps tiny until the moment estimates stabilize. This is the motivation behind both linear warmup and the RAdam "rectified Adam" variant. The canonical transformer schedule is linear warmup → cosine (or inverse-sqrt) decay.

The original Transformer used warmup followed by inverse-square-root decay:

ηt=dmodel−0.5⋅min⁡ ⁣(t−0.5,  t⋅Tw−1.5)\eta_t = d_\text{model}^{-0.5} \cdot \min\!\left(t^{-0.5},\; t \cdot T_w^{-1.5}\right)

which ramps linearly to a peak at t=Twt = T_w, then decays as t−0.5t^{-0.5}.

Adaptive and cyclic schedules

Reduce-on-plateau

Rather than a fixed calendar, watch a validation metric and cut η\eta (e.g. ×0.1) whenever it stops improving for patience epochs. Robust when you don't know the right schedule in advance; reacts to the actual training dynamics.

One-cycle

Anneal η\eta up to a large maximum over the first ~40% of training, then back down below the start, often with momentum moving inversely. The high-rate middle phase acts as regularization ("super-convergence") and can train some models in far fewer epochs.

Warm restarts (SGDR)

Run cosine annealing to the floor, then jump η\eta back up and anneal again, repeating with progressively longer cycles. Each restart kicks the optimizer out of a sharp minimum; the set of models captured just before each restart also makes a cheap ensemble (snapshot ensembling).

A practical chooser

SituationSchedule
Transformers / LLM pretraininglinear warmup → cosine (or inverse-sqrt) decay
LLM fine-tuningshort linear warmup → linear decay to 0
CNN from scratchstep decay, or one-cycle for speed
Don't know the right schedulereduce-on-plateau on val loss
Want a cheap ensemblecosine warm restarts (SGDR) + snapshots

The schedule length must match the run

Cosine and linear decay are parameterized by the total number of steps TT. If you set TT longer than you actually train, η\eta never reaches its floor and the model is under-annealed; if shorter, it sits at the floor doing nothing for the tail. Always set TT to the real planned step count — a common silent bug when training is cut short or extended.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

RelatedReferenced by
Learning Rate Schedules: Warmup, …Gradient Descent Optimize…Gradient DescentGradient Descent VariantsTraining Techniques & Hyp…