Training Techniques & Hyperparameter SearchOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Training Techniques & Hyperparameter Search

A model's architecture is only one part of the story. The learning rate schedule, when to stop training, how aggressively to search the hyperparameter space, and what data transformations to apply during training often determine whether you get 80% or 95% accuracy.

Early stopping

The problem: neural networks can overfit as training continues — training loss keeps decreasing while validation loss plateaus and then rises.

Early stopping: monitor validation loss after each epoch; stop training when validation loss hasn't improved for patience consecutive epochs; restore the weights from the best validation epoch.

Python
best_val_loss = float('inf')
best_weights  = None
patience_count = 0

for epoch in range(max_epochs):
    train_model(model)
    val_loss = evaluate(model, val_data)

    if val_loss < best_val_loss:
        best_val_loss  = val_loss
        best_weights   = copy_weights(model)
        patience_count = 0
    else:
        patience_count += 1
        if patience_count >= patience:
            break

restore_weights(model, best_weights)

Worked example: training a network on a noisy dataset. Optimal stopping is at epoch 50 — after that, the model memorizes training-set noise.

EpochTrain lossVal loss
100.800.82
300.550.57
500.400.42 (best)
800.200.55
1000.100.70

Regularization equivalence: for gradient descent on quadratic loss, early stopping is exactly equivalent to L2 regularization — the number of training steps plays the role of 1/λ1/\lambda.

Patience, priced

What patience actually buys

0.20.40.60.81.01.2restore epoch 22stop at 28trainvalidationepoch
epochs run (this curve)28
epochs run (800-run mean)33.4
restored loss (800-run mean)0.4196
best achievable0.4058
regret vs the true minimum0.0138

One seeded run of a 60-epoch training: training loss falls forever, validation loss turns around. The teal band is the patience window counting down from the best epoch; the vertical line is where training stops and the marker is the checkpoint that gets restored. The right-hand statistics are averages over 800 independent noise draws of the same underlying curve, so no single lucky run is doing the arguing.

Run the rule 800 times against the same underlying curve with fresh noise and the advice becomes a table. The best achievable validation loss here is 0.40580.4058, at epoch 33:

patienceepochs runrestored lossregret
113.20.63880.2329
221.00.49000.0841
326.20.44700.0412
533.40.41960.0138
840.10.41210.0062
1246.20.41050.0047
2054.60.41070.0048

Patience 1 is a disaster — a single noisy uptick ends the run 20 epochs early and costs 0.230.23 of loss. But look at the other end: past patience 8 the regret is flat at noise level, and the only thing more patience buys is more epochs. That asymmetry is the actual rule. Because early stopping restores the best checkpoint rather than the last one, over-large patience wastes compute and nothing else, while too-small patience loses quality you cannot get back. When in doubt, err large.

Setting patience

Patience of 5–20 epochs is typical. Too small → stop before the model has converged past a plateau. Too large → waste compute waiting for a recovery that won't come. For transformers, 3–5 validations is often enough; for CNNs, 10–20 epochs.

Learning rate schedules

A fixed learning rate is rarely optimal throughout training — the standard schedules and when to reach for each:

ScheduleBest forKey parameter
Step decayResNets, CNNsγ and M
Cosine annealingGeneral deep learningT, η_min
Warm-up + cosineTransformers, BERT, ViTwarmup_steps
Reduce on plateauWhen you don't know Tpatience, factor

Warm-up is essential for Transformers trained with Adam: at t=1t=1, Adam's second-moment estimate v^t≈0\hat{v}_t \approx 0, making the effective step size unstable — a linear ramp over the first few hundred steps lets the optimizer accumulate reliable statistics first.

The schedule formulas, SGDR warm restarts, and how each interacts with the optimizer are covered in Gradient Descent Variants and the deep dive:

Deep dive
Learning Rate Schedules: Warmup, Decay & Restarts
Includes runnable notebook

Hyperparameter search, briefly

Three strategies in increasing sophistication — grid (exhaustive, exponential in dimensions), random (beats grid because each of nn trials samples a fresh value of every hyperparameter, so the dimensions that matter get nn distinct values instead of nk\sqrt[k]{n}), and Bayesian optimization (a Gaussian-Process surrogate plus an acquisition function like Expected Improvement steers each expensive evaluation toward promising-or-uncertain regions; typically 3–5× fewer runs than random). Successive halving / Hyperband add early stopping to the search itself: start many configs cheaply, keep the best half, double their budget, repeat.

Two rules survive every strategy: sample the learning rate and regularization strength on a log scale (their effects are multiplicative), and score configurations with cross-validation on training data only.

The full treatment (search-space design, when each method wins, and the AutoML/NAS landscape) is Hyperparameter Optimization & AutoML.

Data augmentation

Data augmentation creates new training examples by applying label-preserving transformations to existing ones:

Image augmentation:

  • Random crops (crop 224×224 from a 256×256 image)
  • Horizontal flip (label-invariant for most tasks)
  • Color jitter (brightness, contrast, saturation, hue)
  • Mixup: x~=λxi+(1−λ)xj\tilde{x} = \lambda x_i + (1-\lambda) x_j, y~=λyi+(1−λ)yj\tilde{y} = \lambda y_i + (1-\lambda) y_j
  • CutMix: replace rectangular patches with pixels from another image

Text augmentation: synonym replacement, back-translation, random deletion/swap.

Regularization effect: augmentation forces the model to learn features invariant to the applied transforms. It's equivalent to enlarging the training set and preventing memorization of specific pixel values.

Apply augmentation only during training

Augmentation is applied only to the training set. Applying random crops or flips at test time (except for Test-Time Augmentation / TTA where you average predictions over multiple augmented copies) would introduce noise and reduce accuracy.

Test-Time Augmentation (TTA): at inference, apply multiple augmentations to each test image and average the predictions. This can add 0.5–2% accuracy at the cost of k× inference time.

Exercise

Early stopping monitors validation loss during training and stops when it stops decreasing. What regularization effect does this have?

Need a hint?

Think about what early stopping prevents the optimizer from doing to the weights.

Exercise

Transformer models typically use a learning rate warm-up phase followed by decay. Why is warm-up necessary for Adam with Transformers?

Need a hint?

Think about Adam's second moment estimate at t=1 vs. t=1000.

Exercise

A model has 2 hyperparameters: learning rate (important) and batch size (relatively unimportant). You can afford 25 evaluations. Why does random search typically beat grid search here?

Need a hint?

In grid search, how many distinct values does each hyperparameter get when you use 25 evaluations?

Common mistakes

  1. Monitoring training loss instead of validation loss for early stopping: training loss always decreases; you need validation loss to detect overfitting.
  2. Over-tuning on the test set: if you run 100 hyperparameter configurations and select the one with the best test score, you've effectively fitted the test set.
  3. Fixed patience too small: some models plateau for 10-20 epochs before improving; 2-3 patience can stop training prematurely.
  4. Forgetting augmentation at test time (accidentally): if your data loader applies augmentation by default, make sure to switch to eval mode or disable augmentation during testing.
  5. Grid search with linearly spaced LR values: learning rate effects are multiplicative; always use log-spaced grids.

Key takeaways

  • Early stopping monitors validation loss and restores best weights; it is equivalent to L2 regularization for quadratic losses.
  • Learning rate schedules (cosine annealing, warm-up + decay) improve convergence; warm-up is essential for Transformers.
  • Random search outperforms grid search when some hyperparameters matter more than others — it covers more distinct values per important dimension.
  • Bayesian optimization uses a GP surrogate and acquisition function to focus evaluations on promising regions; best for expensive training runs.
  • Data augmentation enforces label-invariant transforms during training, acting as regularization and preventing memorization.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelatedGo deeper
Training Techniques & Hyperparame…Linear & Logistic Regress…KNN & Decision TreesNeural NetworksValidation StrategiesGradient Descent VariantsHyperparameter Optimizati…Deployment PitfallsLearning Rate Schedules: …