Linear RegressionOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Linear Regression

The simplest supervised learning task: given input features x1,…,xdx_1, \ldots, x_d, predict a continuous output yy. It is also the cleanest first trip around the ML project loop — the linear model is the whole hypothesis space, and least squares is the objective.

Deep dive
The ML Project Loop
Includes runnable notebook

The forecasting thread

The site's running example (the daily-demand forecasting problem from the project-loop hub) starts right here. Its tabular baseline is exactly this model: build features from the recent past (yt−1y_{t-1}, yt−7y_{t-7}) and the calendar (day-of-week), then fit them with ordinary least squares. Every later course revisits the same problem with a richer hypothesis space; this is trip number one.

Real-world examples: Predicting house prices from square footage, bedrooms, and location. Estimating a patient's blood pressure from age and weight. Forecasting sales from advertising spend.

The model

y^=w0+w1x1+w2x2+⋯+wdxd\hat{y} = w_0 + w_1 x_1 + w_2 x_2 + \cdots + w_d x_d

Compactly, y^=w⊤x\hat{y} = \mathbf{w}^\top \mathbf{x} where w=(w0,w1,…,wd)⊤\mathbf{w} = (w_0, w_1, \ldots, w_d)^\top and x=(1,x1,…,xd)⊤\mathbf{x} = (1, x_1, \ldots, x_d)^\top (a leading 11 absorbs the bias). In matrix form across nn examples: y^=Xw\hat{\mathbf{y}} = X\mathbf{w}, where X∈Rn×(d+1)X \in \mathbb{R}^{n \times (d+1)} prepends a column of ones to the feature matrix.

Each weight wjw_j tells you how much yy changes when xjx_j increases by one unit, holding all other features constant. This is the key interpretation of linear regression.

Intuition

Imagine plotting house price vs. square footage. Linear regression finds the straight line that best fits the points. The slope is w1w_1 — for each additional square foot, the predicted price increases by w1w_1 dollars.

Ordinary Least Squares (OLS)

We find the weights that minimize the mean squared error (MSE):

L(w)=1n∑i=1n(yi−y^i)2=1n∥y−Xw∥2\mathcal{L}(\mathbf{w}) = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2 = \frac{1}{n} \| \mathbf{y} - X\mathbf{w} \|^2

The 1n\frac{1}{n} is just a positive constant. It does not move the location of the minimum, so to keep the algebra clean we minimize the plain sum of squared errors ∥y−Xw∥2\| \mathbf{y} - X\mathbf{w} \|^2 instead. The minimizer is identical.

The closed-form minimizer is obtained by expanding the squared norm, taking the gradient, setting it to zero, and solving — a four-step algebra exercise using two standard matrix-calculus identities. The result:

  w∗=(X⊤X)−1X⊤y  \boxed{\;\mathbf{w}^* = (X^\top X)^{-1} X^\top \mathbf{y}\;}

(X⊤X)−1(X^\top X)^{-1} exists when the feature columns are linearly independent; Ridge regression adds λI\lambda I to guarantee invertibility. For the full derivation and a complete 3-point hand fit (including the 2×22\times2 inversion and R2R^2 calculation), see:

Deep dive
OLS: The Normal Equation
Includes runnable notebook

Why closed-form matters

Unlike gradient descent, OLS gives the exact answer in one computation — no learning rate to tune, no convergence to monitor. The cost is an O(d3)O(d^3) matrix inversion, which becomes slow when dd is large.

Residuals and R2R^2 for this fit

Using the three-point dataset {(1,2), (2,3), (3,5)}\{(1,2),\,(2,3),\,(3,5)\} (also used in the Python code below) (where the normal equation gives w=(w0,w1)=(0.333, 1.5)\mathbf{w} = (w_0, w_1) = (0.333,\,1.5)) let's verify the fit quality. A residual is the gap between truth and prediction, ri=yi−y^ir_i = y_i - \hat{y}_i. Plug each xix_i into the fitted line:

y^1=0.333+1.5⋅1=1.833,y^2=0.333+1.5⋅2=3.333,y^3=0.333+1.5⋅3=4.833\hat{y}_1 = 0.333 + 1.5\cdot1 = 1.833, \quad \hat{y}_2 = 0.333 + 1.5\cdot2 = 3.333, \quad \hat{y}_3 = 0.333 + 1.5\cdot3 = 4.833 r1=2−1.833=0.167,r2=3−3.333=−0.333,r3=5−4.833=0.167r_1 = 2 - 1.833 = 0.167, \quad r_2 = 3 - 3.333 = -0.333, \quad r_3 = 5 - 4.833 = 0.167

The residual sum of squares measures leftover error:

SSres=∑iri2=0.1672+(−0.333)2+0.1672≈0.0278+0.111+0.0278=0.167\text{SS}_{\text{res}} = \sum_i r_i^2 = 0.167^2 + (-0.333)^2 + 0.167^2 \approx 0.0278 + 0.111 + 0.0278 = 0.167

The total sum of squares measures how much yy varies around its mean yˉ=2+3+53=3.333\bar{y} = \frac{2+3+5}{3} = 3.333:

SStot=∑i(yi−yˉ)2=(2−3.333)2+(3−3.333)2+(5−3.333)2≈1.778+0.111+2.778=4.667\text{SS}_{\text{tot}} = \sum_i (y_i - \bar{y})^2 = (2-3.333)^2 + (3-3.333)^2 + (5-3.333)^2 \approx 1.778 + 0.111 + 2.778 = 4.667

The coefficient of determination R2R^2 is the fraction of variance the model explains:

R2=1−SSresSStot=1−0.1674.667≈0.964R^2 = 1 - \frac{\text{SS}_{\text{res}}}{\text{SS}_{\text{tot}}} = 1 - \frac{0.167}{4.667} \approx 0.964

So this line explains about 96% of the variation in yy. R2=1R^2 = 1 is a perfect fit; R2=0R^2 = 0 means the model does no better than always predicting the mean yˉ\bar{y}.

Least squares: minimise the residuals

xy
MSE2.546
min MSE0.335

MSE is the mean of the squared red segments. No setting of slope/intercept beats the OLS solution — that's what 'least squares' means.

Python implementation

Python
import numpy as np

# Data: a leading column of ones encodes the bias w0
X = np.array([[1, 1], [1, 2], [1, 3]])
y = np.array([2, 3, 5])

# Closed-form normal equation: w = (X^T X)^-1 X^T y
w = np.linalg.inv(X.T @ X) @ X.T @ y
print("Weights [w0, w1]:", w.round(3))   # [0.333 1.5]

# Predictions and residuals
y_hat = X @ w
resid = y - y_hat
print("Residuals:", resid.round(3))      # [ 0.167 -0.333  0.167]

# R^2 = 1 - SS_res / SS_tot
ss_res = (resid ** 2).sum()
ss_tot = ((y - y.mean()) ** 2).sum()
print("R^2:", round(1 - ss_res / ss_tot, 3))  # 0.964

# Predict a new point x = 4
x_new = np.array([1, 4])  # bias=1, feature=4
print("Prediction at x=4:", round(x_new @ w, 3))  # 6.333

The problem: overfitting

With many features (or correlated features), XTXX^T X can be nearly singular, causing the weights to blow up. The model fits the training noise perfectly but generalizes poorly.

Signs of overfitting in linear regression:

  • Training error is much lower than test error
  • Weight magnitudes are very large
  • Small changes in data cause wild weight swings

The fix: regularization

Both symptoms — huge unstable weights and a (nearly) singular X⊤XX^\top X — have the same cure: penalize weight magnitude so the model must earn its complexity. The Ridge penalty λ∥w∥2\lambda\|\mathbf{w}\|^2 even repairs the algebra: its gradient adds 2λw2\lambda\mathbf{w} to the normal equation, so the closed form becomes

w∗=(X⊤X+λI)−1X⊤y\mathbf{w}^* = (X^\top X + \lambda I)^{-1} X^\top \mathbf{y}

and the λI\lambda I term lifts every eigenvalue of X⊤XX^\top X above zero — the numerical fix and the statistical fix are the same fix.

How Ridge (L2) and Lasso (L1) differ (why L1 zeroes features out entirely while L2 only shrinks them, and how to choose λ\lambda) is the subject of Regularization: Ridge & Lasso.

Exercise

What is the closed-form solution for Ordinary Least Squares regression?

Need a hint?

It involves the inverse of X^T X.

Exercise

Given X = [[1,1],[1,2],[1,3]] and y = [2,3,5], what is w* for OLS?

Need a hint?

Compute w* = (X^T X)^{-1} X^T y.

Common mistakes

  1. Comparing raw weight magnitudes across features: a weight's size depends on its feature's scale; standardize features before reading wjw_j as importance.
  2. Evaluating on training data alone: a high training R2R^2 with many features says little; check held-out error to catch overfitting.
  3. Assuming linear means simple: Linear regression with polynomial features, interactions, and transforms can model complex relationships while remaining interpretable.

Key takeaways

  • Linear regression fits y=w⊤xy = \mathbf{w}^\top\mathbf{x} by minimizing squared error.
  • OLS has a closed form: w∗=(X⊤X)−1X⊤y\mathbf{w}^* = (X^\top X)^{-1}X^\top\mathbf{y}.
  • R2R^2 measures the fraction of variance explained; residuals show where the fit misses.
  • Overfitting shows up as huge, unstable weights — the cure is regularization, covered in Ridge & Lasso.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelatedGo deeper
Linear RegressionLinear Algebra for MLCalculus for MLGradient DescentRegularization: Ridge & L…Logistic RegressionBagging & Random ForestsRecurrent Neural NetworksTime Series FundamentalsThe ML Project LoopOLS: The Normal Equation