Linear Regression
The simplest supervised learning task: given input features , predict a continuous output . It is also the cleanest first trip around the ML project loop — the linear model is the whole hypothesis space, and least squares is the objective.
The forecasting thread
The site's running example (the daily-demand forecasting problem from the project-loop hub) starts right here. Its tabular baseline is exactly this model: build features from the recent past (, ) and the calendar (day-of-week), then fit them with ordinary least squares. Every later course revisits the same problem with a richer hypothesis space; this is trip number one.
Real-world examples: Predicting house prices from square footage, bedrooms, and location. Estimating a patient's blood pressure from age and weight. Forecasting sales from advertising spend.
The model
Compactly, where and (a leading absorbs the bias). In matrix form across examples: , where prepends a column of ones to the feature matrix.
Each weight tells you how much changes when increases by one unit, holding all other features constant. This is the key interpretation of linear regression.
Intuition
Imagine plotting house price vs. square footage. Linear regression finds the straight line that best fits the points. The slope is — for each additional square foot, the predicted price increases by dollars.
Ordinary Least Squares (OLS)
We find the weights that minimize the mean squared error (MSE):
The is just a positive constant. It does not move the location of the minimum, so to keep the algebra clean we minimize the plain sum of squared errors instead. The minimizer is identical.
The closed-form minimizer is obtained by expanding the squared norm, taking the gradient, setting it to zero, and solving — a four-step algebra exercise using two standard matrix-calculus identities. The result:
exists when the feature columns are linearly independent; Ridge regression adds to guarantee invertibility. For the full derivation and a complete 3-point hand fit (including the inversion and calculation), see:
Why closed-form matters
Unlike gradient descent, OLS gives the exact answer in one computation — no learning rate to tune, no convergence to monitor. The cost is an matrix inversion, which becomes slow when is large.
Residuals and for this fit
Using the three-point dataset (also used in the Python code below) (where the normal equation gives ) let's verify the fit quality. A residual is the gap between truth and prediction, . Plug each into the fitted line:
The residual sum of squares measures leftover error:
The total sum of squares measures how much varies around its mean :
The coefficient of determination is the fraction of variance the model explains:
So this line explains about 96% of the variation in . is a perfect fit; means the model does no better than always predicting the mean .
Least squares: minimise the residuals
MSE is the mean of the squared red segments. No setting of slope/intercept beats the OLS solution — that's what 'least squares' means.
Python implementation
import numpy as np
# Data: a leading column of ones encodes the bias w0
X = np.array([[1, 1], [1, 2], [1, 3]])
y = np.array([2, 3, 5])
# Closed-form normal equation: w = (X^T X)^-1 X^T y
w = np.linalg.inv(X.T @ X) @ X.T @ y
print("Weights [w0, w1]:", w.round(3)) # [0.333 1.5]
# Predictions and residuals
y_hat = X @ w
resid = y - y_hat
print("Residuals:", resid.round(3)) # [ 0.167 -0.333 0.167]
# R^2 = 1 - SS_res / SS_tot
ss_res = (resid ** 2).sum()
ss_tot = ((y - y.mean()) ** 2).sum()
print("R^2:", round(1 - ss_res / ss_tot, 3)) # 0.964
# Predict a new point x = 4
x_new = np.array([1, 4]) # bias=1, feature=4
print("Prediction at x=4:", round(x_new @ w, 3)) # 6.333
The problem: overfitting
With many features (or correlated features), can be nearly singular, causing the weights to blow up. The model fits the training noise perfectly but generalizes poorly.
Signs of overfitting in linear regression:
- Training error is much lower than test error
- Weight magnitudes are very large
- Small changes in data cause wild weight swings
The fix: regularization
Both symptoms — huge unstable weights and a (nearly) singular — have the same cure: penalize weight magnitude so the model must earn its complexity. The Ridge penalty even repairs the algebra: its gradient adds to the normal equation, so the closed form becomes
and the term lifts every eigenvalue of above zero — the numerical fix and the statistical fix are the same fix.
How Ridge (L2) and Lasso (L1) differ (why L1 zeroes features out entirely while L2 only shrinks them, and how to choose ) is the subject of Regularization: Ridge & Lasso.
Exercise
What is the closed-form solution for Ordinary Least Squares regression?
Need a hint?
It involves the inverse of X^T X.
Exercise
Given X = [[1,1],[1,2],[1,3]] and y = [2,3,5], what is w* for OLS?
Need a hint?
Compute w* = (X^T X)^{-1} X^T y.
Common mistakes
- Comparing raw weight magnitudes across features: a weight's size depends on its feature's scale; standardize features before reading as importance.
- Evaluating on training data alone: a high training with many features says little; check held-out error to catch overfitting.
- Assuming linear means simple: Linear regression with polynomial features, interactions, and transforms can model complex relationships while remaining interpretable.
Key takeaways
- Linear regression fits by minimizing squared error.
- OLS has a closed form: .
- measures the fraction of variance explained; residuals show where the fit misses.
- Overfitting shows up as huge, unstable weights — the cure is regularization, covered in Ridge & Lasso.
Related concepts
- Regularization: Ridge & Lasso: the cure for the overfitting problem raised here
- Logistic Regression: the classification cousin of the same linear model
- Gradient Descent: the iterative alternative to the closed-form solution
- Bagging & Random Forests: non-linear models when a line isn't enough