Learning Rate Schedules: Warmup, Decay & Restarts
The learning rate is the single most important training hyperparameter, but the best value changes over the course of training. Early on you want large steps to traverse the loss landscape quickly; near a minimum you want small steps to settle without bouncing. A learning rate schedule varies across steps to get both, and modern recipes (especially for transformers) treat the schedule as essential, not optional.
Why a constant rate is a compromise
A fixed forces one value to serve two conflicting phases. Too large for the whole run and training oscillates around the minimum, never settling — the loss plateaus high. Too small and early training crawls, wasting compute and risking getting stuck in a poor region. Decaying the rate over time resolves the tension: big steps while far away, small steps to fine-tune. Empirically, a well-chosen decay schedule reaches a lower final loss than any constant rate.
Decay schedules
Let be the base rate and the step (or epoch).
Step decay
Drop the rate by a factor every steps:
Simple and a long-standing default for vision (e.g. ×0.1 at epochs 30, 60, 90). The discontinuous drops are visible as sudden loss improvements.
Exponential decay
Smooth version of step decay: . No cliffs, one decay-rate knob.
Cosine annealing
The modern default. Anneal along a half-cosine from down to a floor \eta_\min over steps:
\eta_t = \eta_\min + \tfrac{1}{2}(\eta_0 - \eta_\min)\left(1 + \cos\frac{\pi t}{T}\right)It spends a long time at high rates early, then decays gently — the shape that pairs well with warmup for transformers and large-batch training.
Polynomial / linear decay
. With this is the linear decay-to-zero used in many LLM fine-tuning recipes.
Warmup
Starting at the full learning rate can be destabilizing — especially for adaptive optimizers and large batches. Warmup ramps up from ~0 to over the first steps (usually linearly), then hands off to a decay schedule:
Why Adam needs warmup
Adam divides each step by a running estimate of the gradient's second moment. In the first few steps that estimate is computed from almost no data, so it is high-variance — early updates can be wildly mis-scaled and shove the weights into a bad region. Warmup keeps steps tiny until the moment estimates stabilize. This is the motivation behind both linear warmup and the RAdam "rectified Adam" variant. The canonical transformer schedule is linear warmup → cosine (or inverse-sqrt) decay.
The original Transformer used warmup followed by inverse-square-root decay:
which ramps linearly to a peak at , then decays as .
Adaptive and cyclic schedules
Reduce-on-plateau
Rather than a fixed calendar, watch a validation metric and cut (e.g. ×0.1) whenever it stops improving for patience epochs. Robust when you don't know the right schedule in advance; reacts to the actual training dynamics.
One-cycle
Anneal up to a large maximum over the first ~40% of training, then back down below the start, often with momentum moving inversely. The high-rate middle phase acts as regularization ("super-convergence") and can train some models in far fewer epochs.
Warm restarts (SGDR)
Run cosine annealing to the floor, then jump back up and anneal again, repeating with progressively longer cycles. Each restart kicks the optimizer out of a sharp minimum; the set of models captured just before each restart also makes a cheap ensemble (snapshot ensembling).
A practical chooser
| Situation | Schedule |
|---|---|
| Transformers / LLM pretraining | linear warmup → cosine (or inverse-sqrt) decay |
| LLM fine-tuning | short linear warmup → linear decay to 0 |
| CNN from scratch | step decay, or one-cycle for speed |
| Don't know the right schedule | reduce-on-plateau on val loss |
| Want a cheap ensemble | cosine warm restarts (SGDR) + snapshots |
The schedule length must match the run
Cosine and linear decay are parameterized by the total number of steps . If you set longer than you actually train, never reaches its floor and the model is under-annealed; if shorter, it sits at the floor doing nothing for the tail. Always set to the real planned step count — a common silent bug when training is cut short or extended.
Related concepts
- Gradient Descent: the update the learning rate scales, and why its size controls speed vs stability
- Training Techniques: schedules alongside early stopping, regularization, and batch size
- Gradient Descent Optimizers: the optimizer the schedule sits on top of; AdamW + cosine is the standard pairing