Principal Component AnalysisOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Principal Component Analysis (PCA)

PCA finds the directions of maximum variance in the data and projects onto them. It's the most popular dimensionality reduction technique. In the ML project loop it lives in the data slot: it's a representation transform applied before modelling, changing what the raw material looks like rather than what the model or the loss is.

Deep dive
The ML Project Loop
Includes runnable notebook

Real-world examples: Data visualization (plotting high-dimensional data in 2D), noise reduction, feature preprocessing, image compression, exploratory data analysis.

The core idea

Given dd-dimensional data, PCA finds k≪dk \ll d orthogonal directions (principal components) that capture the most variance:

PC1=arg⁡max⁡∥v∥=1Var(Xv)=direction of maximum variance\text{PC1} = \arg\max_{\|\mathbf{v}\|=1} \text{Var}(\mathbf{Xv}) = \text{direction of maximum variance} PC2=direction of max variance, perpendicular to PC1\text{PC2} = \text{direction of max variance, perpendicular to PC1} ⋮\vdots

Each subsequent component captures the most remaining variance while being orthogonal to all previous components.

Intuition

Imagine looking at a cloud of points shaped like a flat pancake in 3D. PCA finds the two directions that span the pancake's face. You can describe each point's location with just 2 coordinates instead of 3. The third dimension (thickness) doesn't carry much information.

PCA: the axis of maximum variance

feature x₁feature x₂
variance captured55.8%
status—

Each grey segment is a point's projection onto the axis. PC1 is the direction that captures the most variance — rotate the axis or hit 'Find PC1' to land on it.

The algorithm

Step 1: Center the data

X~=X−Xˉ\tilde{X} = X - \bar{X}

Subtract the mean from each feature (PCA is sensitive to scale, not mean — standardize if features have different units or vastly different ranges).

Step 2: Compute covariance matrix

C=1n−1X~TX~C = \frac{1}{n-1} \tilde{X}^T \tilde{X}

The covariance matrix captures how features vary together.

Step 3: Eigendecomposition

C=VΛVTC = V \Lambda V^T
  • VV: eigenvectors (principal component directions)
  • Λ\Lambda: diagonal matrix of eigenvalues (variance explained by each component)

Step 4: Project onto top kk components

Xreduced=X~VkX_{\text{reduced}} = \tilde{X} V_k

where VkV_k contains the first kk eigenvectors.

Why is PC1 the top eigenvector?

This is the heart of PCA. The variance of the data projected onto a unit direction v\mathbf{v} is

Var(X~v)=1n−1(X~v)⊤(X~v)=v⊤1n−1X~⊤X~⏟C v=v⊤C v.\text{Var}(\tilde{X}\mathbf{v}) = \frac{1}{n-1}(\tilde{X}\mathbf{v})^\top(\tilde{X}\mathbf{v}) = \mathbf{v}^\top \underbrace{\tfrac{1}{n-1}\tilde{X}^\top\tilde{X}}_{C}\,\mathbf{v} = \mathbf{v}^\top C\,\mathbf{v}.

PC1 maximizes this subject to ∥v∥=1\|\mathbf{v}\|=1. Use a Lagrange multiplier λ\lambda:

L(v,λ)=v⊤Cv−λ(v⊤v−1),∂L∂v=2Cv−2λv=0.\mathcal{L}(\mathbf{v},\lambda) = \mathbf{v}^\top C \mathbf{v} - \lambda(\mathbf{v}^\top\mathbf{v} - 1), \qquad \frac{\partial \mathcal{L}}{\partial \mathbf{v}} = 2C\mathbf{v} - 2\lambda\mathbf{v} = \mathbf{0}. ⇒  Cv=λv.\Rightarrow\; C\mathbf{v} = \lambda\mathbf{v}.

So the optimum v\mathbf{v} is an eigenvector of CC. Left-multiplying by v⊤\mathbf{v}^\top gives the variance it captures: v⊤Cv=λ v⊤v=λ\mathbf{v}^\top C \mathbf{v} = \lambda\, \mathbf{v}^\top\mathbf{v} = \lambda. To maximize variance, pick the eigenvector with the largest eigenvalue — that's PC1. PC2 is the next-largest (automatically orthogonal, since CC is symmetric), and so on. The eigenvalue is the variance along that component.

Python
# The idiom (for your own feature matrix X):
import numpy as np
from sklearn.decomposition import PCA

pca = PCA(n_components=2)
X_reduced = pca.fit_transform(X)

print(f"Explained variance: {pca.explained_variance_ratio_}")
print(f"Total variance explained: {sum(pca.explained_variance_ratio_):.3f}")

Worked example (by hand)

Take 4 centered 2D points: (2,1), (1,2), (−1,−2), (−2,−1)(2,1),\,(1,2),\,(-1,-2),\,(-2,-1) (mean is already 0\mathbf 0).

Step 2: covariance C=1n−1X~⊤X~C = \tfrac{1}{n-1}\tilde X^\top \tilde X with n=4n=4:

∑xi2=4+1+1+4=10,∑yi2=10,∑xiyi=2+2+2+2=8\textstyle\sum x_i^2 = 4{+}1{+}1{+}4 = 10,\quad \sum y_i^2 = 10,\quad \sum x_i y_i = 2{+}2{+}2{+}2 = 8 C=13[108810]=[3.332.672.673.33].C = \tfrac{1}{3}\begin{bmatrix} 10 & 8 \\ 8 & 10 \end{bmatrix} = \begin{bmatrix} 3.33 & 2.67 \\ 2.67 & 3.33 \end{bmatrix}.

Step 3: eigenvalues via the characteristic polynomial λ2−(tr C)λ+det⁡C=0\lambda^2 - (\text{tr}\,C)\lambda + \det C = 0 (same trick as the eigenvalues lesson). Here tr C=6.67\text{tr}\,C = 6.67 and det⁡C=3.332−2.672=11.11−7.11=4.0\det C = 3.33^2 - 2.67^2 = 11.11 - 7.11 = 4.0:

λ2−6.67λ+4.0=0  ⇒  λ1=6.0,λ2=0.667.\lambda^2 - 6.67\lambda + 4.0 = 0 \;\Rightarrow\; \lambda_1 = 6.0,\quad \lambda_2 = 0.667.

Eigenvectors: solving (C−λI)v=0(C-\lambda I)\mathbf v = \mathbf 0 gives v1∝(1,1)\mathbf v_1 \propto (1,1) for λ1\lambda_1 and v2∝(1,−1)\mathbf v_2 \propto (1,-1) for λ2\lambda_2 — the data stretches most along the 45°45° diagonal.

Variance explained: λ1λ1+λ2=6.06.667=90%\dfrac{\lambda_1}{\lambda_1+\lambda_2} = \dfrac{6.0}{6.667} = 90\%. Keeping only PC1 (the (1,1)(1,1) direction) retains 90% of the variance while halving the dimensions.

Python
import numpy as np

X = np.array([[2,1],[1,2],[-1,-2],[-2,-1]])   # already centered
C = np.cov(X.T)                                # (1/(n-1)) X^T X
eigvals, eigvecs = np.linalg.eigh(C)           # ascending order
print("covariance:\n", C)
print("eigenvalues:", np.round(eigvals[::-1], 3))          # [6.0, 0.667]
print("variance ratio:", np.round(eigvals[::-1]/eigvals.sum(), 3))  # [0.9, 0.1]

Choosing the number of components

The eigenvalues rank the components, but how many should you keep? The standard answers — cumulative explained variance thresholds, the scree elbow, and (most principled) cross-validating the downstream model — plus reconstruction error and whitening, are the subject of PCA in Practice.

SVD connection

In practice, PCA is computed via Singular Value Decomposition (SVD), which is numerically more stable:

X~=UΣVT\tilde{X} = U \Sigma V^T
  • VV: principal component directions (same as eigenvectors of CC)
  • Σ\Sigma: singular values (related to eigenvalues by λi=si2/(n−1)\lambda_i = s_i^2 / (n-1))
Python
# Manual PCA via SVD
X_centered = X - X.mean(axis=0)
U, S, Vt = np.linalg.svd(X_centered, full_matrices=False)

# Vt rows are principal components
X_reduced = X_centered @ Vt[:2].T  # project onto first 2 PCs

Applications

Data visualization

Project high-dimensional data to 2D for plotting:

Python
from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler

iris = load_iris()
X_scaled = StandardScaler().fit_transform(iris.data)  # (150, 4)

pca = PCA(n_components=2)
X_2d = pca.fit_transform(X_scaled)

plt.figure(figsize=(8, 6))
for label in [0, 1, 2]:
    mask = iris.target == label
    plt.scatter(X_2d[mask, 0], X_2d[mask, 1], label=iris.target_names[label], alpha=0.7)
plt.xlabel(f'PC1 ({pca.explained_variance_ratio_[0]:.1%})')
plt.ylabel(f'PC2 ({pca.explained_variance_ratio_[1]:.1%})')
plt.legend()
plt.title('Iris dataset — PCA projection')
plt.show()

Noise reduction and preprocessing

Projecting to fewer components and reconstructing (inverse_transform) denoises data, and feeding the projection to a model shrinks its input dimension — both workflows, including the fit-on-train-only rule and reconstruction error, are covered in PCA in Practice.

Limitations

✅ Strengths❌ Weaknesses
Fast, well-understoodOnly captures linear relationships
Removes multicollinearityMaximizes variance, not class separability
Reduces noiseSensitive to outliers
No hyperparametersRequires feature scaling
Foundational techniqueComponents are hard to interpret

PCA maximizes variance, not accuracy

PCA finds directions of maximum variance, but these may not be the best for classification. If you want class-discriminative projections, use Linear Discriminant Analysis (LDA) instead.

Exercise

PCA finds the directions that maximize what property?

Need a hint?

Think about what PCA preserves when reducing dimensions.

Exercise

If the first 3 principal components explain 85% of variance, what does the remaining 15% represent?

Need a hint?

Think about what information is lost.

Exercise

Why must you standardize features before PCA?

Need a hint?

PCA maximizes variance.

Common mistakes

  1. Not standardizing features: PCA is sensitive to feature scales. A feature with range [0, 1000] will dominate over [0, 1]. Always standardize.
  2. Using PCA before train/test split: Fit PCA on training data only, then transform both train and test. Otherwise you leak information.
  3. Keeping too many components: If you keep all components, PCA does nothing. The whole point is reduction.

Key takeaways

  • PCA finds orthogonal directions (eigenvectors of the covariance) of maximum variance.
  • Projecting onto the top kk components reduces dimensions while keeping most variance.
  • Choose kk from cumulative explained variance (e.g. 95%) or the scree elbow.
  • Standardize first; PCA is linear and unsupervised, computed via SVD.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelatedGo deeper
Principal Component AnalysisClusteringLinear Algebra for MLProbability & StatisticsMatrices and Transformati…Eigenvalues and Eigenvect…SVD & Low-Rank Approximat…K-Nearest NeighborsThe Kernel TrickK-Means ClusteringThe ML Project LoopFactor AnalysisIndependent Component Ana…Computing Eigenvalues & E…