Ridge & Lasso Regularization Paths
Both Ridge and Lasso add a penalty to OLS to prevent large weights, but the choice of penalty norm leads to fundamentally different behavior: Ridge shrinks everything smoothly, Lasso zeros features out exactly.
The closed-form Ridge (L2) solution
Ridge adds a squared-norm penalty:
Step 1 — expand the squared norm:
Step 2 — take the gradient with respect to :
Step 3 — set to zero:
Step 4 — solve:
The term does two things simultaneously:
- Invertibility: may be singular (collinear features), but has eigenvalues — always invertible.
- Shrinkage: every weight is pulled toward zero proportionally. In the eigenbasis of with eigenvalues , Ridge multiplies each component by — an attenuation that shrinks small-variance directions most aggressively.
Ridge = MAP with Gaussian prior
Putting a zero-mean Gaussian prior and taking the MAP estimate recovers exactly the Ridge solution with . A tighter prior (smaller ) corresponds to stronger regularization (larger ).
Lasso (L1) sparsity without a closed form
Lasso adds an absolute-value penalty:
There is no closed-form solution — the absolute value is non-differentiable at . Instead, we use subdifferential calculus: the subdifferential of at zero is the interval , so the optimality condition becomes:
The second condition is the key: if the gradient of the data-fit term is small enough (magnitude ), the weight stays exactly zero and never re-enters the model. The L1 penalty applies a constant force of magnitude toward zero regardless of ; any feature whose signal is weaker than this force gets zeroed out.
Why L2 doesn't zero things out: Ridge applies a force of magnitude — proportional to the weight. As , the force also goes to zero, so the weight asymptotically approaches but never reaches zero.
Watch the zeros happen
The conditions above say when a weight stays at zero. Coordinate descent is what actually enforces them, and it is one line — cycle over the coordinates, and set each to where is that feature's correlation with the partial residual:
Lasso by coordinate descent — where the exact zeros come from
Six standardized features, forty rows, and the objective (1/2n)‖y − Xw‖² + λ‖w‖₁ with λ = 0.25. Standardizing so that (1/n)Σxⱼ² = 1 is what makes the coordinate update collapse to a single line later. Three features carry real signal (true weights 3, 1.2, 0.4); x₃ and x₄ are pure noise; **x₅ is a near-copy of x₀** — correlation 0.967 — with a true weight of zero. Start from w = 0, so the residual is just y and every ρⱼ is the plain correlation of a column with the target.
| true w | corr(xⱼ, y) | role | |
|---|---|---|---|
| x₀ | 3 | 2.9153 | signal |
| x₁ | 1.2 | 1.3002 | signal |
| x₂ | 0.4 | 0.5537 | signal |
| x₃ | 0 | 0.5064 | noise |
| x₄ | 0 | 0.7004 | noise |
| x₅ | 0 | 2.8906 | echo of x₀ |
Look at the noise rows before going on: x₃ and x₄ correlate 0.51 and 0.70 with y — both comfortably above λ = 0.25. On marginal correlation alone they would survive.
This page derives the condition that keeps a weight at zero but never runs the algorithm that enforces it. Coordinate descent does it in one line: w_j = soft(ρ_j, λ), where the max(·, 0) inside soft is the only thing on the page capable of producing an exact zero — watch it fire on the two noise features in sweep 1, and compare against Ridge's ρ/(1+λ), which is never flat. Three measured results complicate the page's summary: both noise features have marginal correlations above λ and are killed only by the partial residual; the correlated echo has a true weight of zero yet outlives two genuinely relevant features; and the same data with that correlation removed converges in 5 sweeps instead of 267.
What to notice:
- The
max(·, 0)is the entire source of sparsity. Nothing else in the procedure can produce a zero. Compare it against Ridge's coordinate update in the same plot: a straight line, never flat, never zero. - Marginal correlation is not what kills a feature. Both noise columns correlate 0.51 and 0.70 with (well above ) and would survive on that alone. They die because by the time the loop reaches them, the features already fitted have absorbed the signal and their partial-residual has collapsed to ~0.08.
- "Weakest features die first" is not quite right. The correlated echo has a true weight of zero yet outlives both genuinely relevant features, because Lasso ranks by correlation with the residual, not relevance to the target. A zero from Lasso means "added nothing given the others", not "irrelevant", which is the argument for elastic net, and why Lasso's selections are unstable across bootstrap resamples.
- Correlation is expensive. The same data converges in 267 sweeps with the echo present and 5 without it: the correlated pair passes the shared signal back and forth one small step at a time.
The geometric picture of L1 vs L2
Picture the constraint region in weight space:
- L2 (sphere): is a circle (in 2D). The loss contours expand until they touch the circle. A circle has no corners — the touching point is almost never on an axis.
- L1 (diamond): is a diamond. Its corners lie on the axes (where one or more ). Elliptical loss contours almost always touch the diamond at a corner — producing a sparse solution.
Coefficient path evolution as λ varies
As increases from 0 to :
- Ridge: all coefficients shrink smoothly, asymptotically approaching 0 — never exactly 0.
- Lasso: coefficients reach exactly 0 at different λ values (the weaker features die first), producing a piecewise-linear path.
import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import Ridge, Lasso
from sklearn.preprocessing import StandardScaler
np.random.seed(42)
# Synthetic 10-feature dataset: 3 strong, 2 medium, 2 correlated, 3 noise
n, p = 200, 10
X = np.random.randn(n, p)
X[:, 7] = X[:, 0] + 0.2 * np.random.randn(n) # correlated echo
true_w = np.array([3, 2, 1.5, 0, 0, 0.8, 0.5, 0, 0, 0])
y = X @ true_w + np.random.randn(n)
scaler = StandardScaler()
X_s = scaler.fit_transform(X)
lambdas = np.logspace(-3, 2, 100)
coefs_ridge = np.array([Ridge(alpha=lam).fit(X_s, y).coef_ for lam in lambdas])
coefs_lasso = np.array([Lasso(alpha=lam, max_iter=5000).fit(X_s, y).coef_ for lam in lambdas])
colors = ['#6366f1','#20d9d2','#f97316','#ef4444','#a78bfa',
'#34d399','#fbbf24','#fb7185','#60a5fa','#94a3b8']
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(13, 5))
for j in range(p):
ax1.semilogx(lambdas, coefs_ridge[:, j], color=colors[j], lw=1.5,
label=f'w{j}' if true_w[j] != 0 else None)
ax2.semilogx(lambdas, coefs_lasso[:, j], color=colors[j], lw=1.5)
ax1.set_title('Ridge paths: smooth shrinkage, never zero')
ax1.set_xlabel('λ (log scale)'); ax1.set_ylabel('Coefficient')
ax1.axhline(0, color='#555', lw=0.5); ax1.grid(True, alpha=0.3)
ax2.set_title('Lasso paths: features die out exactly')
ax2.set_xlabel('λ (log scale)')
ax2.axhline(0, color='#555', lw=0.5); ax2.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()
Lasso's feature elimination order
The order in which Lasso zeros features depends on the correlation between each feature and the current residual:
from sklearn.linear_model import lasso_path
# Lasso path (exact, piecewise-linear)
alphas, coefs_path, _ = lasso_path(X_s, y, n_alphas=200)
# Find the λ at which each coefficient first goes to zero
for j in range(p):
dead_at = alphas[np.where(coefs_path[j] == 0)[0][0]] if np.any(coefs_path[j] == 0) else None
print(f"w{j} (true={true_w[j]:.1f}): zeroed at λ={dead_at:.4f}" if dead_at else f"w{j}: never zeroed")
Strong features (high ) survive to larger λ; noise features die first. This is the path algorithm's key insight: as λ increases, features drop in order of decreasing relevance.
Choosing λ with cross-validation
from sklearn.linear_model import RidgeCV, LassoCV
# Ridge: analytically efficient LOO CV
ridge_cv = RidgeCV(alphas=np.logspace(-3, 3, 100), cv=5)
ridge_cv.fit(X_s, y)
print(f"Ridge best λ: {ridge_cv.alpha_:.4f}")
# Lasso: K-fold CV
lasso_cv = LassoCV(n_alphas=100, cv=5, max_iter=5000, random_state=42)
lasso_cv.fit(X_s, y)
print(f"Lasso best λ: {lasso_cv.alpha_:.4f}")
print(f"Lasso nonzero features: {np.sum(lasso_cv.coef_ != 0)}")
Related concepts
- Regularization: Ridge & Lasso: overview, geometric picture, and choosing between them
- OLS: The Normal Equation: the unregularized baseline that Ridge extends
- Beta-Bernoulli Posterior: Bayesian view: Ridge = Gaussian prior, Lasso = Laplace prior