Regularization: Ridge & Lasso
A model with enough parameters can fit any training set perfectly — including its noise. Linear regression with many correlated features is especially prone to this: weights blow up to huge opposing values that cancel on the training data but swing wildly on anything new. Regularization fixes this by making large weights expensive — a change to the objective slot: you add a penalty term to the loss, leaving the model family and the optimizer alone.
Real-world examples: Genomics (thousands of gene features, few patients), house-price models with dozens of correlated features, any setting where features outnumber samples.
The regularized objective
Instead of minimizing the squared error alone, we add a penalty on the weights:
The hyperparameter sets the exchange rate between the two goals:
- — ordinary least squares; nothing restrains the weights
- — the penalty dominates; all weights are crushed to zero and the model predicts the mean
The two classic choices of differ by one exponent, and behave completely differently.
Ridge regression (L2)
Squaring means the penalty's gradient is — proportional to the weight itself. Big weights feel strong pressure, small weights barely any. The result: Ridge shrinks all weights smoothly toward zero but almost never exactly to zero.
Ridge also has a closed-form solution: . Adding makes the matrix invertible even when features are perfectly correlated — the numerical fix and the statistical fix are the same fix. For the 4-step algebraic derivation, the Lasso sparsity proof via subdifferential calculus, and coefficient path plots across λ:
Why Ridge loves correlated features
Given two nearly identical features, OLS may assign them weights +1000 and −998. Ridge instead splits the weight between them (≈ +1 each): the sum of squares of two halves is smaller than the square of one whole.
Lasso (L1)
The absolute value's gradient is — a constant force regardless of the weight's size. A weight whose contribution to fitting the data is worth less than gets pushed all the way to exactly zero and stays there.
This is why Lasso performs feature selection: as you raise , features drop out of the model one by one, leaving a sparse, interpretable subset.
Watch it happen. Below, six weights are re-fit at every : three carry real signal, one is a correlated echo, two are pure noise. Switch between Ridge and Lasso and drag — Ridge shrinks everything smoothly, Lasso snaps the useless weights to exactly zero first:
Regularization paths: how λ shrinks the weights
Each curve is one weight, re-fit at every λ. Ridge (L2) shrinks all weights smoothly but they stay non-zero forever. Lasso (L1) snaps weights to exactly zero one at a time — the noise weights die first, then the weakest signals: built-in feature selection.
The geometric picture
Picture the constraint regions: L2 is a circle, L1 is a diamond. The loss contours expand until they touch the region, and they almost always touch the diamond at a corner, where some coordinates are exactly zero. Circles have no corners, so Ridge solutions are never sparse.
Choosing between them — and λ
| Situation | Reach for |
|---|---|
| Many features, most probably irrelevant | Lasso: prunes them |
| Many correlated, all somewhat relevant | Ridge: shares weight gracefully |
| Want both behaviors | Elastic Net: a blend: |
itself is chosen by cross-validation: fit at a grid of values, pick the one with the best held-out error. Plotting validation error against reproduces the classic U-shape — underfitting on the right, overfitting on the left, the sweet spot in between. This is the same bias–variance trade-off you'll meet in the bias–variance lesson, expressed as a single dial.
Exercise
You train Ridge regression and see training error 0.1, validation error 2.4. What should you do with λ?
Need a hint?
A big train/validation gap is a variance problem.
Exercise
Which regularization method can set feature weights exactly to zero?
Need a hint?
Think about the shape of the constraint region.
Common mistakes
- Regularizing unstandardized features: the penalty treats all weights alike, so a feature measured in large units gets an unfairly small weight. Standardize first.
- Penalizing the intercept: the bias term shouldn't pay the complexity tax; exclude it from (libraries do this by default).
- Tuning on the test set: choose by cross-validation on training data; the test set is touched once, at the end.
Key takeaways
- Regularization adds a weight penalty so the model must earn its complexity
- Ridge (L2) shrinks weights smoothly; great with correlated features; never sparse
- Lasso (L1) zeroes out weak features entirely — built-in feature selection
- trades bias for variance; tune it with cross-validation, never on training error
Related concepts
- Linear Regression: the OLS model these penalties tame
- Bias-Variance Tradeoff: the trade-off dials
- Bayesian Inference: Ridge is MAP with a Gaussian prior; Lasso, a Laplace prior