PCA in PracticeOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

PCA in Practice

The PCA lesson gave you the theory: components are eigenvectors of the covariance matrix, ranked by explained variance. This lesson is about using it well — how many components to keep, what reconstruction error means, and the preprocessing details that quietly make or break results.

How many components?

Three standard answers, in increasing order of principle:

  1. Cumulative explained variance. Keep enough components to cover a target (90%, 95%, 99%). Plot the cumulative ratio ∑i≤mλi/∑iλi\sum_{i \le m} \lambda_i / \sum_i \lambda_i against mm and read off the threshold.
  2. The scree elbow. Plot eigenvalues in decreasing order; keep components before the curve flattens into the "rubble" of noise-level eigenvalues.
  3. Downstream performance. If PCA feeds a classifier, choose mm by cross-validating the whole pipeline. The variance a component explains is not the same as the usefulness of that component — a low-variance direction can carry the class signal.

Fit PCA inside the pipeline

Standardize and fit PCA on the training fold only, then apply the learned transform to the validation fold. Fitting PCA on all data before splitting leaks information and inflates your scores — one of the most common silent bugs in ML pipelines.

Reconstruction: compression with a quality dial

Projecting to mm components and mapping back gives the best rank-mm approximation of your data:

x^=μ+∑i=1m(vi⊤(x−μ)) vi\hat{\mathbf{x}} = \boldsymbol{\mu} + \sum_{i=1}^{m} (\mathbf{v}_i^\top (\mathbf{x} - \boldsymbol{\mu}))\, \mathbf{v}_i

The mean squared reconstruction error equals exactly the sum of the discarded eigenvalues ∑i>mλi\sum_{i > m} \lambda_i — variance kept is fidelity kept. This is PCA as lossy compression: store mm coefficients per sample instead of dd features.

The classic demonstration is eigenfaces: run PCA on thousands of face images and the leading components are ghostly face-like patterns — lighting direction, face width, glasses. Any face compresses to ~50–150 coefficients with recognizable reconstruction, from inputs of thousands of pixels. Reconstruction error also doubles as a simple anomaly detector: a sample that reconstructs poorly doesn't live in the subspace the training data occupies.

Here is that equation as a picture — toggle how many components survive and watch where the error goes:

Reconstruction from m components

feature x₁feature x₂
— PC1 axis- - PC2 axis— reconstruction error● reconstructed point
components kept1 / 2
variance explained88.5%
reconstruction MSE0.285
discarded λ₂0.285

With m=1, every point is replaced by its shadow on PC1 (teal axis); the rose segments are the information thrown away, and their mean square equals exactly the discarded eigenvalue λ₂. With m=2, nothing is discarded and reconstruction is perfect — that's all of PCA's compression story in one picture.

Whitening

Standard PCA output has uncorrelated coordinates with unequal variances (λ1≫λm\lambda_1 \gg \lambda_m). Whitening additionally rescales each coordinate by 1/λi1/\sqrt{\lambda_i}, producing features with identity covariance — uncorrelated and unit variance.

Useful when a downstream algorithm is sensitive to relative scales (distance-based methods, some neural-net inputs). The cost: directions that were mostly noise get amplified to the same size as real signal, so whitening with too many components can hurt. Keep fewer, whiten, validate.

When PCA is the wrong tool

PCA is linear and variance-chasing. It will not unroll a swiss-roll manifold, and it can bury a low-variance class boundary. For visualization of non-linear structure, hand the data to t-SNE or UMAP — typically after a PCA down to ~50 dimensions, which removes noise and makes the neighbor computations tractable. The two tools are teammates, not rivals.

Exercise

PCA feeds a classifier. The first 10 components explain 95% of variance, but validation accuracy peaks at 40 components. How many should you keep?

Need a hint?

Variance explained is PCA's objective — is it the classifier's?

Key takeaways

  • Choose mm by cumulative variance for compression, by cross-validation when PCA feeds a model
  • Reconstruction error = discarded eigenvalues; it powers compression and simple anomaly detection
  • Whitening equalizes component variances — useful for scale-sensitive consumers, risky on noise components
  • Fit PCA on training data only; for non-linear structure, PCA first, then t-SNE/UMAP

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelatedGo deeper
PCA in PracticeClusteringLinear Algebra for MLProbability & StatisticsSVD & Low-Rank Approximat…Principal Component Analy…t-SNE & UMAPSVD and the Eckart-Young …