Regularization: Ridge & LassoOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Regularization: Ridge & Lasso

A model with enough parameters can fit any training set perfectly — including its noise. Linear regression with many correlated features is especially prone to this: weights blow up to huge opposing values that cancel on the training data but swing wildly on anything new. Regularization fixes this by making large weights expensive — a change to the objective slot: you add a penalty term to the loss, leaving the model family and the optimizer alone.

Real-world examples: Genomics (thousands of gene features, few patients), house-price models with dozens of correlated features, any setting where features outnumber samples.

The regularized objective

Instead of minimizing the squared error alone, we add a penalty on the weights:

L(w)=∑i=1n(yi−w⊤xi)2⏟fit the data+λ⋅Ω(w)⏟stay simple\mathcal{L}(\mathbf{w}) = \underbrace{\sum_{i=1}^{n} (y_i - \mathbf{w}^\top \mathbf{x}_i)^2}_{\text{fit the data}} + \lambda \cdot \underbrace{\Omega(\mathbf{w})}_{\text{stay simple}}

The hyperparameter λ≥0\lambda \geq 0 sets the exchange rate between the two goals:

  • λ=0\lambda = 0 — ordinary least squares; nothing restrains the weights
  • λ→∞\lambda \to \infty — the penalty dominates; all weights are crushed to zero and the model predicts the mean

The two classic choices of Ω\Omega differ by one exponent, and behave completely differently.

Ridge regression (L2)

Ω(w)=∥w∥22=∑jwj2\Omega(\mathbf{w}) = \|\mathbf{w}\|_2^2 = \sum_j w_j^2

Squaring means the penalty's gradient is 2λwj2\lambda w_j — proportional to the weight itself. Big weights feel strong pressure, small weights barely any. The result: Ridge shrinks all weights smoothly toward zero but almost never exactly to zero.

Ridge also has a closed-form solution: w∗=(X⊤X+λI)−1X⊤y\mathbf{w}^* = (X^\top X + \lambda I)^{-1} X^\top \mathbf{y}. Adding λI\lambda I makes the matrix invertible even when features are perfectly correlated — the numerical fix and the statistical fix are the same fix. For the 4-step algebraic derivation, the Lasso sparsity proof via subdifferential calculus, and coefficient path plots across λ:

Deep dive
Ridge & Lasso Regularization Paths
Includes runnable notebook

Why Ridge loves correlated features

Given two nearly identical features, OLS may assign them weights +1000 and −998. Ridge instead splits the weight between them (≈ +1 each): the sum of squares of two halves is smaller than the square of one whole.

Lasso (L1)

Ω(w)=∥w∥1=∑j∣wj∣\Omega(\mathbf{w}) = \|\mathbf{w}\|_1 = \sum_j |w_j|

The absolute value's gradient is ±λ\pm\lambda — a constant force regardless of the weight's size. A weight whose contribution to fitting the data is worth less than λ\lambda gets pushed all the way to exactly zero and stays there.

This is why Lasso performs feature selection: as you raise λ\lambda, features drop out of the model one by one, leaving a sparse, interpretable subset.

Watch it happen. Below, six weights are re-fit at every λ\lambda: three carry real signal, one is a correlated echo, two are pure noise. Switch between Ridge and Lasso and drag λ\lambda — Ridge shrinks everything smoothly, Lasso snaps the useless weights to exactly zero first:

Regularization paths: how λ shrinks the weights

0.010.1110100-4-2024λ (log scale)weight value
— w₁ (signal +4)— w₂ (signal −3)— w₃ (signal +2)— w₄ (corr. w/ x₁)— w₅ (noise)— w₆ (noise)
λ1.00
non-zero weights3 / 6
‖w‖₁5.04

Each curve is one weight, re-fit at every λ. Ridge (L2) shrinks all weights smoothly but they stay non-zero forever. Lasso (L1) snaps weights to exactly zero one at a time — the noise weights die first, then the weakest signals: built-in feature selection.

The geometric picture

Picture the constraint regions: L2 is a circle, L1 is a diamond. The loss contours expand until they touch the region, and they almost always touch the diamond at a corner, where some coordinates are exactly zero. Circles have no corners, so Ridge solutions are never sparse.

Choosing between them — and λ

SituationReach for
Many features, most probably irrelevantLasso: prunes them
Many correlated, all somewhat relevantRidge: shares weight gracefully
Want both behaviorsElastic Net: a blend: α∥w∥1+(1−α)∥w∥22\alpha\|\mathbf{w}\|_1 + (1-\alpha)\|\mathbf{w}\|_2^2

λ\lambda itself is chosen by cross-validation: fit at a grid of values, pick the one with the best held-out error. Plotting validation error against λ\lambda reproduces the classic U-shape — underfitting on the right, overfitting on the left, the sweet spot in between. This is the same bias–variance trade-off you'll meet in the bias–variance lesson, expressed as a single dial.

Exercise

You train Ridge regression and see training error 0.1, validation error 2.4. What should you do with λ?

Need a hint?

A big train/validation gap is a variance problem.

Exercise

Which regularization method can set feature weights exactly to zero?

Need a hint?

Think about the shape of the constraint region.

Common mistakes

  1. Regularizing unstandardized features: the penalty treats all weights alike, so a feature measured in large units gets an unfairly small weight. Standardize first.
  2. Penalizing the intercept: the bias term shouldn't pay the complexity tax; exclude it from Ω(w)\Omega(\mathbf{w}) (libraries do this by default).
  3. Tuning λ\lambda on the test set: choose λ\lambda by cross-validation on training data; the test set is touched once, at the end.

Key takeaways

  • Regularization adds a weight penalty so the model must earn its complexity
  • Ridge (L2) shrinks weights smoothly; great with correlated features; never sparse
  • Lasso (L1) zeroes out weak features entirely — built-in feature selection
  • λ\lambda trades bias for variance; tune it with cross-validation, never on training error

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelatedGo deeper
Regularization: Ridge & LassoLinear Algebra for MLCalculus for MLBatchNorm & DropoutLinear RegressionThe Bias–Variance TradeoffBayesian InferenceLearning Theory & General…Feature EngineeringRidge & Lasso Regularizat…Bias-Variance Decompositi…OLS: The Normal EquationVC Dimension