Ordinary Least Squares (OLS) minimizes the sum of squared residuals ∥y−Xw∥2 and has an exact closed-form solution. This page derives that solution step by step and verifies it on a concrete 3-point example.
The OLS objective
We want the weight vector w that minimizes:
L(w)=∥y−Xw∥2=(y−Xw)⊤(y−Xw)
The n1 scaling constant is dropped — it doesn't affect where the minimum is.
Deriving the normal-equation solution
Step 1: Expand the squared norm. Use (a−b)⊤(a−b)=a⊤a−2a⊤b+b⊤b (valid because y⊤Xw is a scalar, equal to its transpose w⊤X⊤y):
L(w)=y⊤y−2w⊤X⊤y+w⊤X⊤Xw
Step 2: Take the gradient with respect to w. Two standard matrix-calculus identities:
∇w(w⊤b)=b,∇w(w⊤Aw)=2Aw(A symmetric)
A=X⊤X is symmetric. The first term y⊤y has no w, so its gradient is 0:
∇wL=−2X⊤y+2X⊤Xw
Step 3: Set the gradient to zero. This is the normal equation:
−2X⊤y+2X⊤Xw=0⟹X⊤Xw=X⊤y
Step 4: Solve for w. Left-multiply both sides by (X⊤X)−1:
w∗=(X⊤X)−1X⊤y
This is a global minimum (not a saddle point) because the Hessian ∇w2L=2X⊤X is positive semi-definite — the loss is convex.
When does (XᵀX)⁻¹ exist?
X⊤X is invertible if and only if the columns of X are linearly independent — no feature is a linear combination of the others (no perfect collinearity) and n≥d (not more features than samples). When this fails, use Ridge regression: (X⊤X+λI) is always invertible for λ>0.
A hand-worked 3-point example
Data: (x,y)=(1,2),(2,3),(3,5). Fit y^=w0+w1x.
Add a leading ones column for the bias w0:
X=111123,y=235
Compute X⊤X (a 2×2 matrix):
X⊤X=[1+1+11+2+31+2+31+4+9]=[36614]
Compute X⊤y:
X⊤y=[2+3+51⋅2+2⋅3+3⋅5]=[1023]
Invert X⊤X. For [acbd]−1=ad−bc1[d−c−ba]. Determinant =3⋅14−62=42−36=6: