Hyperparameter Optimization & AutoMLOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Hyperparameter Optimization & AutoML

Gradient descent learns the parameters (weights). But the hyperparameters (learning rate, regularization strength, layer width, tree depth) are set before training and can't be learned by the same gradient. Choosing them well is often the difference between a great model and a mediocre one, and doing it efficiently is its own optimization problem. This lesson covers how to search hyperparameter space, and the AutoML vision of automating the search entirely.

Real-world examples: tuning a model's learning rate and weight decay, choosing tree depth and number of estimators for XGBoost, neural architecture search, and managed AutoML services that try to deliver a good model with minimal manual tuning.

Why hyperparameters are different

Parameters are optimized inside training by gradient descent; hyperparameters are knobs outside it. You can't backprop through "use 3 layers vs 4" or "learning rate 0.01 vs 0.1" — they're often discrete, the objective (validation score) is expensive (a full training run) and noisy, and you have no gradient. That's exactly the expensive, black-box, derivative-free setting — the same shape as the Bayesian-optimization problem.

λ∗=arg⁡max⁡λ  ValidationScore(train(λ))\boldsymbol{\lambda}^{*} = \arg\max_{\boldsymbol{\lambda}} \; \text{ValidationScore}\big(\text{train}(\boldsymbol{\lambda})\big)

Each evaluation of the objective is a complete training run, so the whole game is finding good λ\boldsymbol{\lambda} in few evaluations.

The search strategies

Grid search. Try every combination on a predefined grid. Exhaustive and simple, but it scales exponentially with the number of hyperparameters (the curse of dimensionality) and wastes trials on dimensions that don't matter.

Random search. Sample configurations at random. Surprisingly, it usually beats grid search for the same budget: most hyperparameters barely matter, so random sampling explores the important ones more finely instead of wasting trials on a fine grid over irrelevant ones (Bergstra & Bengio).

Why random beats grid

A grid spends its budget evenly across all dimensions, including useless ones, so it only tries a few distinct values of the one hyperparameter that matters. Random search tries a different value of every hyperparameter on every trial, effectively giving the important dimension many more distinct samples for the same number of runs.

Bayesian optimization. Both grid and random are non-adaptive — they ignore everything learned from past trials. Bayesian optimization (from the Bayesian Methods course) fits a surrogate to past results and picks the most promising next configuration, finding good hyperparameters in far fewer trials. The catch: it's sequential, so it parallelizes less easily than random search.

The same budget, spent two ways

Grid spends its budget on the wrong axis

grid 5×5 — 5 values of xbest 0.7565random — 25 values of xbest 0.9998x →
evaluations25
grid best0.7565
random best (this seed)0.9998
random best (2000-seed mean)0.9797
optimum1.0500

Walk the budget from 9 to 36 and watch the grid column: 0.5171 at 9, 1.0472 at 16, 0.7565 at 25, 0.9843 at 36. It is not monotone — going from 16 evaluations to 25 made it worse, because whether a grid line lands near the peak is an accident of the spacing, and a finer grid can straddle a peak the coarser one hit. Random goes 0.8554 → 0.9402 → 0.9797 → 0.9985, because every extra sample is a fresh draw of the coordinate that matters.

The objective is bright where it is high: x moves it a lot, y almost not at all. Teal dots are the evaluations; the yellow ticks below each panel are their x coordinates — the only coordinate that matters. A k×k grid costs k² evaluations and buys k distinct values of x.

The objective is bright where it is high, and only xx moves it — the case Bergstra & Bengio built the argument on. A k×kk\times k grid costs k2k^2 evaluations and buys kk distinct values of xx; nn random points buy nn. The yellow ticks under each panel are the xx coordinates actually tried, and the difference in coverage is the whole story.

Now sweep the budget, because the interesting failure is not the average:

evaluationsgridrandom (2000-seed mean)
90.51710.8554
161.04720.9402
250.75650.9797
360.98430.9985

At 16 the grid wins outright — one of its lines happens to fall near x=0.62x = 0.62. Then at 25 it gets worse, from 1.04721.0472 down to 0.75650.7565, on 9 more evaluations. Grid search is not monotone in budget: whether a grid line lands on a peak is an accident of the spacing, and refining the grid can straddle a peak the coarser one hit. Random is monotone by construction, because every additional sample is a fresh draw of the coordinate that matters.

Paying more and getting less is a worse property than being beaten on average, and it only shows up if you sweep the budget instead of fixing it.

Early-stopping methods: spend compute where it counts

The newest idea: don't run every configuration to completion. Bad configs usually reveal themselves early, so stop the losers and reallocate their budget to promising configs.

  • Successive Halving: start many configs with a small budget (few epochs), keep the top half, double their budget, repeat. Most configs die cheap; the survivors get the compute.
  • Hyperband: runs Successive Halving at several "aggressiveness" levels to hedge between trying many configs briefly and few configs thoroughly.
  • ASHA: an asynchronous, parallel-friendly version of Successive Halving, the practical default at scale.

These combine naturally with random search (random configs, halving allocation) and dramatically cut total compute.

AutoML and neural architecture search

AutoML aims to automate the whole pipeline (feature preprocessing, model choice, and hyperparameters), so a non-expert can get a strong model. Neural Architecture Search (NAS) is the most ambitious slice: search the model architecture itself (which layers, connections, sizes). Early NAS cost thousands of GPU-days; modern weight-sharing and gradient-based NAS made it far cheaper, though for most practitioners a good search over a sensible space still beats fully automated design.

Don't tune on the test set

Hyperparameter search optimizes the validation score; if you tune against the test set you'll overfit it and your reported numbers will be optimistic. With heavy searching even the validation set can be overfit — use nested cross-validation or a held-out final test set touched only once.

Exercise

For the same number of trials, random search usually outperforms grid search on hyperparameters. Why?

Need a hint?

How many distinct values of the *important* hyperparameter does each method try?

Exercise

Successive Halving / Hyperband start many configurations with a small budget, keep the best, and give survivors more budget. What does this exploit?

Need a hint?

How early can you usually tell a config is bad?

Exercise

You search hundreds of hyperparameter configurations and pick the best. On which data should you select, and where do you finally report performance?

Need a hint?

What happens to the set you optimize against?

Common mistakes

  1. Grid search over many hyperparameters. It's exponential and wasteful — prefer random search or Bayesian optimization.
  2. Running every config to completion. Use early-stopping (Hyperband/ASHA) to kill bad configs cheaply.
  3. Tuning on the test set. That overfits it; search on validation, evaluate once on a held-out test set.
  4. Searching the wrong space. Log-scale the learning rate, pick sensible ranges; a bad search space dooms any method.

Key takeaways

  • Hyperparameters are set before training and can't be learned by gradient descent; tuning them is an expensive, black-box, gradient-free optimization over the validation score.
  • Random search usually beats grid search for a fixed budget (it samples the important dimensions more finely); Bayesian optimization is more sample-efficient but sequential.
  • Early-stopping methods (Successive Halving, Hyperband, ASHA) kill bad configs early and reallocate compute, slashing total cost.
  • AutoML/NAS automate model and architecture choice; always tune on validation (use nested CV) and evaluate on a held-out test set once.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelated
Hyperparameter Optimization & Aut…Calculus for MLNeural NetworksGradient Boosting in Prac…Bayesian OptimizationValidation StrategiesGradient Descent VariantsTraining Techniques & Hyp…