Hyperparameter Optimization & AutoML
Gradient descent learns the parameters (weights). But the hyperparameters (learning rate, regularization strength, layer width, tree depth) are set before training and can't be learned by the same gradient. Choosing them well is often the difference between a great model and a mediocre one, and doing it efficiently is its own optimization problem. This lesson covers how to search hyperparameter space, and the AutoML vision of automating the search entirely.
Real-world examples: tuning a model's learning rate and weight decay, choosing tree depth and number of estimators for XGBoost, neural architecture search, and managed AutoML services that try to deliver a good model with minimal manual tuning.
Why hyperparameters are different
Parameters are optimized inside training by gradient descent; hyperparameters are knobs outside it. You can't backprop through "use 3 layers vs 4" or "learning rate 0.01 vs 0.1" — they're often discrete, the objective (validation score) is expensive (a full training run) and noisy, and you have no gradient. That's exactly the expensive, black-box, derivative-free setting — the same shape as the Bayesian-optimization problem.
Each evaluation of the objective is a complete training run, so the whole game is finding good in few evaluations.
The search strategies
Grid search. Try every combination on a predefined grid. Exhaustive and simple, but it scales exponentially with the number of hyperparameters (the curse of dimensionality) and wastes trials on dimensions that don't matter.
Random search. Sample configurations at random. Surprisingly, it usually beats grid search for the same budget: most hyperparameters barely matter, so random sampling explores the important ones more finely instead of wasting trials on a fine grid over irrelevant ones (Bergstra & Bengio).
Why random beats grid
A grid spends its budget evenly across all dimensions, including useless ones, so it only tries a few distinct values of the one hyperparameter that matters. Random search tries a different value of every hyperparameter on every trial, effectively giving the important dimension many more distinct samples for the same number of runs.
Bayesian optimization. Both grid and random are non-adaptive — they ignore everything learned from past trials. Bayesian optimization (from the Bayesian Methods course) fits a surrogate to past results and picks the most promising next configuration, finding good hyperparameters in far fewer trials. The catch: it's sequential, so it parallelizes less easily than random search.
The same budget, spent two ways
Grid spends its budget on the wrong axis
Walk the budget from 9 to 36 and watch the grid column: 0.5171 at 9, 1.0472 at 16, 0.7565 at 25, 0.9843 at 36. It is not monotone — going from 16 evaluations to 25 made it worse, because whether a grid line lands near the peak is an accident of the spacing, and a finer grid can straddle a peak the coarser one hit. Random goes 0.8554 → 0.9402 → 0.9797 → 0.9985, because every extra sample is a fresh draw of the coordinate that matters.
The objective is bright where it is high: x moves it a lot, y almost not at all. Teal dots are the evaluations; the yellow ticks below each panel are their x coordinates — the only coordinate that matters. A k×k grid costs k² evaluations and buys k distinct values of x.
The objective is bright where it is high, and only moves it — the case Bergstra & Bengio built the argument on. A grid costs evaluations and buys distinct values of ; random points buy . The yellow ticks under each panel are the coordinates actually tried, and the difference in coverage is the whole story.
Now sweep the budget, because the interesting failure is not the average:
| evaluations | grid | random (2000-seed mean) |
|---|---|---|
| 9 | 0.5171 | 0.8554 |
| 16 | 1.0472 | 0.9402 |
| 25 | 0.7565 | 0.9797 |
| 36 | 0.9843 | 0.9985 |
At 16 the grid wins outright — one of its lines happens to fall near . Then at 25 it gets worse, from down to , on 9 more evaluations. Grid search is not monotone in budget: whether a grid line lands on a peak is an accident of the spacing, and refining the grid can straddle a peak the coarser one hit. Random is monotone by construction, because every additional sample is a fresh draw of the coordinate that matters.
Paying more and getting less is a worse property than being beaten on average, and it only shows up if you sweep the budget instead of fixing it.
Early-stopping methods: spend compute where it counts
The newest idea: don't run every configuration to completion. Bad configs usually reveal themselves early, so stop the losers and reallocate their budget to promising configs.
- Successive Halving: start many configs with a small budget (few epochs), keep the top half, double their budget, repeat. Most configs die cheap; the survivors get the compute.
- Hyperband: runs Successive Halving at several "aggressiveness" levels to hedge between trying many configs briefly and few configs thoroughly.
- ASHA: an asynchronous, parallel-friendly version of Successive Halving, the practical default at scale.
These combine naturally with random search (random configs, halving allocation) and dramatically cut total compute.
AutoML and neural architecture search
AutoML aims to automate the whole pipeline (feature preprocessing, model choice, and hyperparameters), so a non-expert can get a strong model. Neural Architecture Search (NAS) is the most ambitious slice: search the model architecture itself (which layers, connections, sizes). Early NAS cost thousands of GPU-days; modern weight-sharing and gradient-based NAS made it far cheaper, though for most practitioners a good search over a sensible space still beats fully automated design.
Don't tune on the test set
Hyperparameter search optimizes the validation score; if you tune against the test set you'll overfit it and your reported numbers will be optimistic. With heavy searching even the validation set can be overfit — use nested cross-validation or a held-out final test set touched only once.
Exercise
For the same number of trials, random search usually outperforms grid search on hyperparameters. Why?
Need a hint?
How many distinct values of the *important* hyperparameter does each method try?
Exercise
Successive Halving / Hyperband start many configurations with a small budget, keep the best, and give survivors more budget. What does this exploit?
Need a hint?
How early can you usually tell a config is bad?
Exercise
You search hundreds of hyperparameter configurations and pick the best. On which data should you select, and where do you finally report performance?
Need a hint?
What happens to the set you optimize against?
Common mistakes
- Grid search over many hyperparameters. It's exponential and wasteful — prefer random search or Bayesian optimization.
- Running every config to completion. Use early-stopping (Hyperband/ASHA) to kill bad configs cheaply.
- Tuning on the test set. That overfits it; search on validation, evaluate once on a held-out test set.
- Searching the wrong space. Log-scale the learning rate, pick sensible ranges; a bad search space dooms any method.
Key takeaways
- Hyperparameters are set before training and can't be learned by gradient descent; tuning them is an expensive, black-box, gradient-free optimization over the validation score.
- Random search usually beats grid search for a fixed budget (it samples the important dimensions more finely); Bayesian optimization is more sample-efficient but sequential.
- Early-stopping methods (Successive Halving, Hyperband, ASHA) kill bad configs early and reallocate compute, slashing total cost.
- AutoML/NAS automate model and architecture choice; always tune on validation (use nested CV) and evaluate on a held-out test set once.
Related concepts
- Bayesian Optimization: the sample-efficient, adaptive search method for hyperparameters
- Validation Strategies: the cross-validation that scores each configuration
- Gradient Descent Variants: where the learning rate, a key hyperparameter, lives