Training Techniques & Hyperparameter Search
A model's architecture is only one part of the story. The learning rate schedule, when to stop training, how aggressively to search the hyperparameter space, and what data transformations to apply during training often determine whether you get 80% or 95% accuracy.
Early stopping
The problem: neural networks can overfit as training continues — training loss keeps decreasing while validation loss plateaus and then rises.
Early stopping: monitor validation loss after each epoch; stop training when validation loss hasn't improved for patience consecutive epochs; restore the weights from the best validation epoch.
best_val_loss = float('inf')
best_weights = None
patience_count = 0
for epoch in range(max_epochs):
train_model(model)
val_loss = evaluate(model, val_data)
if val_loss < best_val_loss:
best_val_loss = val_loss
best_weights = copy_weights(model)
patience_count = 0
else:
patience_count += 1
if patience_count >= patience:
break
restore_weights(model, best_weights)
Worked example: training a network on a noisy dataset. Optimal stopping is at epoch 50 — after that, the model memorizes training-set noise.
| Epoch | Train loss | Val loss |
|---|---|---|
| 10 | 0.80 | 0.82 |
| 30 | 0.55 | 0.57 |
| 50 | 0.40 | 0.42 (best) |
| 80 | 0.20 | 0.55 |
| 100 | 0.10 | 0.70 |
Regularization equivalence: for gradient descent on quadratic loss, early stopping is exactly equivalent to L2 regularization — the number of training steps plays the role of .
Patience, priced
What patience actually buys
One seeded run of a 60-epoch training: training loss falls forever, validation loss turns around. The teal band is the patience window counting down from the best epoch; the vertical line is where training stops and the marker is the checkpoint that gets restored. The right-hand statistics are averages over 800 independent noise draws of the same underlying curve, so no single lucky run is doing the arguing.
Run the rule 800 times against the same underlying curve with fresh noise and the advice becomes a table. The best achievable validation loss here is , at epoch 33:
| patience | epochs run | restored loss | regret |
|---|---|---|---|
| 1 | 13.2 | 0.6388 | 0.2329 |
| 2 | 21.0 | 0.4900 | 0.0841 |
| 3 | 26.2 | 0.4470 | 0.0412 |
| 5 | 33.4 | 0.4196 | 0.0138 |
| 8 | 40.1 | 0.4121 | 0.0062 |
| 12 | 46.2 | 0.4105 | 0.0047 |
| 20 | 54.6 | 0.4107 | 0.0048 |
Patience 1 is a disaster — a single noisy uptick ends the run 20 epochs early and costs of loss. But look at the other end: past patience 8 the regret is flat at noise level, and the only thing more patience buys is more epochs. That asymmetry is the actual rule. Because early stopping restores the best checkpoint rather than the last one, over-large patience wastes compute and nothing else, while too-small patience loses quality you cannot get back. When in doubt, err large.
Setting patience
Patience of 5–20 epochs is typical. Too small → stop before the model has converged past a plateau. Too large → waste compute waiting for a recovery that won't come. For transformers, 3–5 validations is often enough; for CNNs, 10–20 epochs.
Learning rate schedules
A fixed learning rate is rarely optimal throughout training — the standard schedules and when to reach for each:
| Schedule | Best for | Key parameter |
|---|---|---|
| Step decay | ResNets, CNNs | γ and M |
| Cosine annealing | General deep learning | T, η_min |
| Warm-up + cosine | Transformers, BERT, ViT | warmup_steps |
| Reduce on plateau | When you don't know T | patience, factor |
Warm-up is essential for Transformers trained with Adam: at , Adam's second-moment estimate , making the effective step size unstable — a linear ramp over the first few hundred steps lets the optimizer accumulate reliable statistics first.
The schedule formulas, SGDR warm restarts, and how each interacts with the optimizer are covered in Gradient Descent Variants and the deep dive:
Hyperparameter search, briefly
Three strategies in increasing sophistication — grid (exhaustive, exponential in dimensions), random (beats grid because each of trials samples a fresh value of every hyperparameter, so the dimensions that matter get distinct values instead of ), and Bayesian optimization (a Gaussian-Process surrogate plus an acquisition function like Expected Improvement steers each expensive evaluation toward promising-or-uncertain regions; typically 3–5× fewer runs than random). Successive halving / Hyperband add early stopping to the search itself: start many configs cheaply, keep the best half, double their budget, repeat.
Two rules survive every strategy: sample the learning rate and regularization strength on a log scale (their effects are multiplicative), and score configurations with cross-validation on training data only.
The full treatment (search-space design, when each method wins, and the AutoML/NAS landscape) is Hyperparameter Optimization & AutoML.
Data augmentation
Data augmentation creates new training examples by applying label-preserving transformations to existing ones:
Image augmentation:
- Random crops (crop 224×224 from a 256×256 image)
- Horizontal flip (label-invariant for most tasks)
- Color jitter (brightness, contrast, saturation, hue)
- Mixup: ,
- CutMix: replace rectangular patches with pixels from another image
Text augmentation: synonym replacement, back-translation, random deletion/swap.
Regularization effect: augmentation forces the model to learn features invariant to the applied transforms. It's equivalent to enlarging the training set and preventing memorization of specific pixel values.
Apply augmentation only during training
Augmentation is applied only to the training set. Applying random crops or flips at test time (except for Test-Time Augmentation / TTA where you average predictions over multiple augmented copies) would introduce noise and reduce accuracy.
Test-Time Augmentation (TTA): at inference, apply multiple augmentations to each test image and average the predictions. This can add 0.5–2% accuracy at the cost of k× inference time.
Exercise
Early stopping monitors validation loss during training and stops when it stops decreasing. What regularization effect does this have?
Need a hint?
Think about what early stopping prevents the optimizer from doing to the weights.
Exercise
Transformer models typically use a learning rate warm-up phase followed by decay. Why is warm-up necessary for Adam with Transformers?
Need a hint?
Think about Adam's second moment estimate at t=1 vs. t=1000.
Exercise
A model has 2 hyperparameters: learning rate (important) and batch size (relatively unimportant). You can afford 25 evaluations. Why does random search typically beat grid search here?
Need a hint?
In grid search, how many distinct values does each hyperparameter get when you use 25 evaluations?
Common mistakes
- Monitoring training loss instead of validation loss for early stopping: training loss always decreases; you need validation loss to detect overfitting.
- Over-tuning on the test set: if you run 100 hyperparameter configurations and select the one with the best test score, you've effectively fitted the test set.
- Fixed patience too small: some models plateau for 10-20 epochs before improving; 2-3 patience can stop training prematurely.
- Forgetting augmentation at test time (accidentally): if your data loader applies augmentation by default, make sure to switch to eval mode or disable augmentation during testing.
- Grid search with linearly spaced LR values: learning rate effects are multiplicative; always use log-spaced grids.
Key takeaways
- Early stopping monitors validation loss and restores best weights; it is equivalent to L2 regularization for quadratic losses.
- Learning rate schedules (cosine annealing, warm-up + decay) improve convergence; warm-up is essential for Transformers.
- Random search outperforms grid search when some hyperparameters matter more than others — it covers more distinct values per important dimension.
- Bayesian optimization uses a GP surrogate and acquisition function to focus evaluations on promising regions; best for expensive training runs.
- Data augmentation enforces label-invariant transforms during training, acting as regularization and preventing memorization.
Related concepts
- Gradient Descent Variants: learning rate schedules applied to optimizers
- Hyperparameter Optimization & AutoML: the full search-strategy treatment
- Validation Strategies: how to avoid leakage during hyperparameter search