Principal Component Analysis (PCA)
PCA finds the directions of maximum variance in the data and projects onto them. It's the most popular dimensionality reduction technique. In the ML project loop it lives in the data slot: it's a representation transform applied before modelling, changing what the raw material looks like rather than what the model or the loss is.
Real-world examples: Data visualization (plotting high-dimensional data in 2D), noise reduction, feature preprocessing, image compression, exploratory data analysis.
The core idea
Given -dimensional data, PCA finds orthogonal directions (principal components) that capture the most variance:
Each subsequent component captures the most remaining variance while being orthogonal to all previous components.
Intuition
Imagine looking at a cloud of points shaped like a flat pancake in 3D. PCA finds the two directions that span the pancake's face. You can describe each point's location with just 2 coordinates instead of 3. The third dimension (thickness) doesn't carry much information.
PCA: the axis of maximum variance
Each grey segment is a point's projection onto the axis. PC1 is the direction that captures the most variance — rotate the axis or hit 'Find PC1' to land on it.
The algorithm
Step 1: Center the data
Subtract the mean from each feature (PCA is sensitive to scale, not mean — standardize if features have different units or vastly different ranges).
Step 2: Compute covariance matrix
The covariance matrix captures how features vary together.
Step 3: Eigendecomposition
- : eigenvectors (principal component directions)
- : diagonal matrix of eigenvalues (variance explained by each component)
Step 4: Project onto top components
where contains the first eigenvectors.
Why is PC1 the top eigenvector?
This is the heart of PCA. The variance of the data projected onto a unit direction is
PC1 maximizes this subject to . Use a Lagrange multiplier :
So the optimum is an eigenvector of . Left-multiplying by gives the variance it captures: . To maximize variance, pick the eigenvector with the largest eigenvalue — that's PC1. PC2 is the next-largest (automatically orthogonal, since is symmetric), and so on. The eigenvalue is the variance along that component.
# The idiom (for your own feature matrix X):
import numpy as np
from sklearn.decomposition import PCA
pca = PCA(n_components=2)
X_reduced = pca.fit_transform(X)
print(f"Explained variance: {pca.explained_variance_ratio_}")
print(f"Total variance explained: {sum(pca.explained_variance_ratio_):.3f}")
Worked example (by hand)
Take 4 centered 2D points: (mean is already ).
Step 2: covariance with :
Step 3: eigenvalues via the characteristic polynomial (same trick as the eigenvalues lesson). Here and :
Eigenvectors: solving gives for and for — the data stretches most along the diagonal.
Variance explained: . Keeping only PC1 (the direction) retains 90% of the variance while halving the dimensions.
import numpy as np
X = np.array([[2,1],[1,2],[-1,-2],[-2,-1]]) # already centered
C = np.cov(X.T) # (1/(n-1)) X^T X
eigvals, eigvecs = np.linalg.eigh(C) # ascending order
print("covariance:\n", C)
print("eigenvalues:", np.round(eigvals[::-1], 3)) # [6.0, 0.667]
print("variance ratio:", np.round(eigvals[::-1]/eigvals.sum(), 3)) # [0.9, 0.1]
Choosing the number of components
The eigenvalues rank the components, but how many should you keep? The standard answers — cumulative explained variance thresholds, the scree elbow, and (most principled) cross-validating the downstream model — plus reconstruction error and whitening, are the subject of PCA in Practice.
SVD connection
In practice, PCA is computed via Singular Value Decomposition (SVD), which is numerically more stable:
- : principal component directions (same as eigenvectors of )
- : singular values (related to eigenvalues by )
# Manual PCA via SVD
X_centered = X - X.mean(axis=0)
U, S, Vt = np.linalg.svd(X_centered, full_matrices=False)
# Vt rows are principal components
X_reduced = X_centered @ Vt[:2].T # project onto first 2 PCs
Applications
Data visualization
Project high-dimensional data to 2D for plotting:
from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler
iris = load_iris()
X_scaled = StandardScaler().fit_transform(iris.data) # (150, 4)
pca = PCA(n_components=2)
X_2d = pca.fit_transform(X_scaled)
plt.figure(figsize=(8, 6))
for label in [0, 1, 2]:
mask = iris.target == label
plt.scatter(X_2d[mask, 0], X_2d[mask, 1], label=iris.target_names[label], alpha=0.7)
plt.xlabel(f'PC1 ({pca.explained_variance_ratio_[0]:.1%})')
plt.ylabel(f'PC2 ({pca.explained_variance_ratio_[1]:.1%})')
plt.legend()
plt.title('Iris dataset — PCA projection')
plt.show()
Noise reduction and preprocessing
Projecting to fewer components and reconstructing (inverse_transform) denoises data, and feeding the projection to a model shrinks its input dimension — both workflows, including the fit-on-train-only rule and reconstruction error, are covered in PCA in Practice.
Limitations
| ✅ Strengths | ❌ Weaknesses |
|---|---|
| Fast, well-understood | Only captures linear relationships |
| Removes multicollinearity | Maximizes variance, not class separability |
| Reduces noise | Sensitive to outliers |
| No hyperparameters | Requires feature scaling |
| Foundational technique | Components are hard to interpret |
PCA maximizes variance, not accuracy
PCA finds directions of maximum variance, but these may not be the best for classification. If you want class-discriminative projections, use Linear Discriminant Analysis (LDA) instead.
Exercise
PCA finds the directions that maximize what property?
Need a hint?
Think about what PCA preserves when reducing dimensions.
Exercise
If the first 3 principal components explain 85% of variance, what does the remaining 15% represent?
Need a hint?
Think about what information is lost.
Exercise
Why must you standardize features before PCA?
Need a hint?
PCA maximizes variance.
Common mistakes
- Not standardizing features: PCA is sensitive to feature scales. A feature with range [0, 1000] will dominate over [0, 1]. Always standardize.
- Using PCA before train/test split: Fit PCA on training data only, then transform both train and test. Otherwise you leak information.
- Keeping too many components: If you keep all components, PCA does nothing. The whole point is reduction.
Key takeaways
- PCA finds orthogonal directions (eigenvectors of the covariance) of maximum variance.
- Projecting onto the top components reduces dimensions while keeping most variance.
- Choose from cumulative explained variance (e.g. 95%) or the scree elbow.
- Standardize first; PCA is linear and unsupervised, computed via SVD.
Related concepts
- t-SNE & UMAP: non-linear methods for visualization
- PCA in Practice: choosing k, reconstruction, whitening
- K-Means Clustering: a common use: reduce, then cluster
- Autoencoders: the non-linear, learned generalization of PCA
- Factor Analysis: the probabilistic latent-variable generalization of PCA (PPCA is the isotropic-noise case)
- Independent Component Analysis: finds independent (not just uncorrelated) components