Ridge & Lasso Regularization PathsOpen in Colab

Ridge & Lasso Regularization Paths

Both Ridge and Lasso add a penalty to OLS to prevent large weights, but the choice of penalty norm leads to fundamentally different behavior: Ridge shrinks everything smoothly, Lasso zeros features out exactly.

The closed-form Ridge (L2) solution

Ridge adds a squared-norm penalty:

LRidge(w)=∥y−Xw∥2+λ∥w∥2\mathcal{L}_\text{Ridge}(\mathbf{w}) = \|\mathbf{y} - X\mathbf{w}\|^2 + \lambda\|\mathbf{w}\|^2

Step 1 — expand the squared norm:

=y⊤y−2w⊤X⊤y+w⊤X⊤Xw+λw⊤w= \mathbf{y}^\top\mathbf{y} - 2\mathbf{w}^\top X^\top\mathbf{y} + \mathbf{w}^\top X^\top X\mathbf{w} + \lambda\mathbf{w}^\top\mathbf{w}

Step 2 — take the gradient with respect to w\mathbf{w}:

∇wL=−2X⊤y+2X⊤Xw+2λw\nabla_\mathbf{w}\mathcal{L} = -2X^\top\mathbf{y} + 2X^\top X\mathbf{w} + 2\lambda\mathbf{w}

Step 3 — set to zero:

−2X⊤y+2(X⊤X+λI)w=0  ⟹  (X⊤X+λI)w=X⊤y-2X^\top\mathbf{y} + 2(X^\top X + \lambda I)\mathbf{w} = \mathbf{0} \;\Longrightarrow\; (X^\top X + \lambda I)\mathbf{w} = X^\top\mathbf{y}

Step 4 — solve:

  wRidge∗=(X⊤X+λI)−1X⊤y  \boxed{\;\mathbf{w}^*_\text{Ridge} = (X^\top X + \lambda I)^{-1} X^\top\mathbf{y}\;}

The λI\lambda I term does two things simultaneously:

  1. Invertibility: X⊤XX^\top X may be singular (collinear features), but X⊤X+λIX^\top X + \lambda I has eigenvalues ≥λ>0\geq\lambda > 0 — always invertible.
  2. Shrinkage: every weight is pulled toward zero proportionally. In the eigenbasis of X⊤XX^\top X with eigenvalues σj2\sigma_j^2, Ridge multiplies each component by σj2/(σj2+λ)\sigma_j^2/(\sigma_j^2+\lambda) — an attenuation that shrinks small-variance directions most aggressively.

Ridge = MAP with Gaussian prior

Putting a zero-mean Gaussian prior w∼N(0,τ2I)\mathbf{w} \sim \mathcal{N}(\mathbf{0}, \tau^2 I) and taking the MAP estimate recovers exactly the Ridge solution with λ=1/(2τ2)\lambda = 1/(2\tau^2). A tighter prior (smaller τ2\tau^2) corresponds to stronger regularization (larger λ\lambda).

Lasso (L1) sparsity without a closed form

Lasso adds an absolute-value penalty:

LLasso(w)=∥y−Xw∥2+λ∥w∥1\mathcal{L}_\text{Lasso}(\mathbf{w}) = \|\mathbf{y} - X\mathbf{w}\|^2 + \lambda\|\mathbf{w}\|_1

There is no closed-form solution — the absolute value is non-differentiable at wj=0w_j = 0. Instead, we use subdifferential calculus: the subdifferential of ∣wj∣|w_j| at zero is the interval [−1,+1][-1, +1], so the optimality condition becomes:

∂∥y−Xw∥2∂wj+λ sign(wj)=0(wj≠0)\frac{\partial \|\mathbf{y}-X\mathbf{w}\|^2}{\partial w_j} + \lambda\,\mathrm{sign}(w_j) = 0 \quad (w_j \neq 0) ∣∂∥y−Xw∥2∂wj∣≤λ(wj=0)\left|\frac{\partial \|\mathbf{y}-X\mathbf{w}\|^2}{\partial w_j}\right| \leq \lambda \quad (w_j = 0)

The second condition is the key: if the gradient of the data-fit term is small enough (magnitude ≤λ\leq\lambda), the weight stays exactly zero and never re-enters the model. The L1 penalty applies a constant force of magnitude λ\lambda toward zero regardless of ∣wj∣|w_j|; any feature whose signal is weaker than this force gets zeroed out.

Why L2 doesn't zero things out: Ridge applies a force of magnitude 2λ∣wj∣2\lambda |w_j| — proportional to the weight. As ∣wj∣→0|w_j| \to 0, the force also goes to zero, so the weight asymptotically approaches but never reaches zero.

Watch the zeros happen

The conditions above say when a weight stays at zero. Coordinate descent is what actually enforces them, and it is one line — cycle over the coordinates, and set each to soft(ρj,λ)=sign(ρj)max⁡(∣ρj∣−λ,0)\mathrm{soft}(\rho_j, \lambda) = \mathrm{sign}(\rho_j)\max(|\rho_j| - \lambda, 0) where ρj\rho_j is that feature's correlation with the partial residual:

Lasso by coordinate descent — where the exact zeros come from

1/9

Six standardized features, forty rows, and the objective (1/2n)‖y − Xw‖² + λ‖w‖₁ with λ = 0.25. Standardizing so that (1/n)Σxⱼ² = 1 is what makes the coordinate update collapse to a single line later. Three features carry real signal (true weights 3, 1.2, 0.4); x₃ and x₄ are pure noise; **x₅ is a near-copy of x₀** — correlation 0.967 — with a true weight of zero. Start from w = 0, so the residual is just y and every ρⱼ is the plain correlation of a column with the target.

python
1def soft(z, lam):
2 # max(-, 0) is the whole source
3 # of sparsity - nothing else zeroes
4 return sign(z) * max(abs(z)-lam, 0)
5
6def lasso_cd(X, y, lam, tol=1e-9):
7 n, p = X.shape
8 w = np.zeros(p)
9 while True:
10 delta = 0.0
11 for j in range(p):
12 r = y - X @ w
13 # partial-residual corr
14 rho = X[:, j] @ r / n + w[j]
15 new = soft(rho, lam)
16 delta = max(delta,
17 abs(new-w[j]))
18 w[j] = new
19 if delta < tol:
20 return w
state
the design
true wcorr(xⱼ, y)role
x₀32.9153signal
x₁1.21.3002signal
x₂0.40.5537signal
x₃00.5064noise
x₄00.7004noise
x₅02.8906echo of x₀

Look at the noise rows before going on: x₃ and x₄ correlate 0.51 and 0.70 with y — both comfortably above λ = 0.25. On marginal correlation alone they would survive.

This page derives the condition that keeps a weight at zero but never runs the algorithm that enforces it. Coordinate descent does it in one line: w_j = soft(ρ_j, λ), where the max(·, 0) inside soft is the only thing on the page capable of producing an exact zero — watch it fire on the two noise features in sweep 1, and compare against Ridge's ρ/(1+λ), which is never flat. Three measured results complicate the page's summary: both noise features have marginal correlations above λ and are killed only by the partial residual; the correlated echo has a true weight of zero yet outlives two genuinely relevant features; and the same data with that correlation removed converges in 5 sweeps instead of 267.

What to notice:

  • The max(·, 0) is the entire source of sparsity. Nothing else in the procedure can produce a zero. Compare it against Ridge's coordinate update ρ/(1+λ)\rho/(1+\lambda) in the same plot: a straight line, never flat, never zero.
  • Marginal correlation is not what kills a feature. Both noise columns correlate 0.51 and 0.70 with yy (well above λ=0.25\lambda = 0.25) and would survive on that alone. They die because by the time the loop reaches them, the features already fitted have absorbed the signal and their partial-residual ρ\rho has collapsed to ~0.08.
  • "Weakest features die first" is not quite right. The correlated echo x5x_5 has a true weight of zero yet outlives both genuinely relevant features, because Lasso ranks by correlation with the residual, not relevance to the target. A zero from Lasso means "added nothing given the others", not "irrelevant", which is the argument for elastic net, and why Lasso's selections are unstable across bootstrap resamples.
  • Correlation is expensive. The same data converges in 267 sweeps with the echo present and 5 without it: the correlated pair passes the shared signal back and forth one small step at a time.

The geometric picture of L1 vs L2

Picture the constraint region in weight space:

  • L2 (sphere): ∥w∥2≤r2\|\mathbf{w}\|^2 \leq r^2 is a circle (in 2D). The loss contours expand until they touch the circle. A circle has no corners — the touching point is almost never on an axis.
  • L1 (diamond): ∥w∥1≤r\|\mathbf{w}\|_1 \leq r is a diamond. Its corners lie on the axes (where one or more wj=0w_j = 0). Elliptical loss contours almost always touch the diamond at a corner — producing a sparse solution.

Coefficient path evolution as λ varies

As λ\lambda increases from 0 to ∞\infty:

  • Ridge: all coefficients shrink smoothly, asymptotically approaching 0 — never exactly 0.
  • Lasso: coefficients reach exactly 0 at different λ values (the weaker features die first), producing a piecewise-linear path.
Python
import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import Ridge, Lasso
from sklearn.preprocessing import StandardScaler

np.random.seed(42)

# Synthetic 10-feature dataset: 3 strong, 2 medium, 2 correlated, 3 noise
n, p = 200, 10
X = np.random.randn(n, p)
X[:, 7] = X[:, 0] + 0.2 * np.random.randn(n)   # correlated echo
true_w = np.array([3, 2, 1.5, 0, 0, 0.8, 0.5, 0, 0, 0])
y = X @ true_w + np.random.randn(n)

scaler = StandardScaler()
X_s = scaler.fit_transform(X)

lambdas = np.logspace(-3, 2, 100)

coefs_ridge = np.array([Ridge(alpha=lam).fit(X_s, y).coef_ for lam in lambdas])
coefs_lasso = np.array([Lasso(alpha=lam, max_iter=5000).fit(X_s, y).coef_ for lam in lambdas])

colors = ['#6366f1','#20d9d2','#f97316','#ef4444','#a78bfa',
          '#34d399','#fbbf24','#fb7185','#60a5fa','#94a3b8']

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(13, 5))

for j in range(p):
    ax1.semilogx(lambdas, coefs_ridge[:, j], color=colors[j], lw=1.5,
                 label=f'w{j}' if true_w[j] != 0 else None)
    ax2.semilogx(lambdas, coefs_lasso[:, j], color=colors[j], lw=1.5)

ax1.set_title('Ridge paths: smooth shrinkage, never zero')
ax1.set_xlabel('λ (log scale)'); ax1.set_ylabel('Coefficient')
ax1.axhline(0, color='#555', lw=0.5); ax1.grid(True, alpha=0.3)

ax2.set_title('Lasso paths: features die out exactly')
ax2.set_xlabel('λ (log scale)')
ax2.axhline(0, color='#555', lw=0.5); ax2.grid(True, alpha=0.3)

plt.tight_layout()
plt.show()

Lasso's feature elimination order

The order in which Lasso zeros features depends on the correlation between each feature and the current residual:

Python
from sklearn.linear_model import lasso_path

# Lasso path (exact, piecewise-linear)
alphas, coefs_path, _ = lasso_path(X_s, y, n_alphas=200)

# Find the λ at which each coefficient first goes to zero
for j in range(p):
    dead_at = alphas[np.where(coefs_path[j] == 0)[0][0]] if np.any(coefs_path[j] == 0) else None
    print(f"w{j} (true={true_w[j]:.1f}): zeroed at λ={dead_at:.4f}" if dead_at else f"w{j}: never zeroed")

Strong features (high ∣corr(xj,r)∣|\text{corr}(x_j, r)|) survive to larger λ; noise features die first. This is the path algorithm's key insight: as λ increases, features drop in order of decreasing relevance.

Choosing λ with cross-validation

Python
from sklearn.linear_model import RidgeCV, LassoCV

# Ridge: analytically efficient LOO CV
ridge_cv = RidgeCV(alphas=np.logspace(-3, 3, 100), cv=5)
ridge_cv.fit(X_s, y)
print(f"Ridge best λ: {ridge_cv.alpha_:.4f}")

# Lasso: K-fold CV
lasso_cv = LassoCV(n_alphas=100, cv=5, max_iter=5000, random_state=42)
lasso_cv.fit(X_s, y)
print(f"Lasso best λ: {lasso_cv.alpha_:.4f}")
print(f"Lasso nonzero features: {np.sum(lasso_cv.coef_ != 0)}")

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

RelatedReferenced by
Ridge & Lasso Regularization PathsOLS: The Normal EquationBeta-Bernoulli Conjugate …Regularization: Ridge & L…