SVD & Low-Rank Approximation
Eigendecomposition was powerful but picky: it needs a square matrix, and the nicest guarantees only arrive when the matrix is symmetric. The singular value decomposition drops every restriction. Any matrix (rectangular, rank-deficient, ugly) factors as:
- — an orthogonal rotation of the input space (columns of = right singular vectors)
- — a diagonal stretch by the singular values
- — an orthogonal rotation of the output space (columns = left singular vectors)
Every linear map, no matter how messy, is just rotate → stretch along perpendicular axes → rotate. The singular values are the stretch factors, and they are always real and non-negative.
Where it comes from
The matrices and are always symmetric, so the spectral theorem — which guarantees symmetric matrices always have real eigenvalues and mutually orthogonal eigenvectors — applies to them even when it doesn't apply to :
- = eigenvectors of
- = eigenvectors of
- , the square roots of their (shared, non-negative) eigenvalues
This is also exactly the PCA connection: for centered data , the covariance is , so the principal components are the right singular vectors of , and explained variances are . Numerical libraries compute PCA via SVD rather than forming the covariance matrix — it's more stable.
The sum-of-layers view and best rank- approximation
Rewriting one column at a time gives a sum of rank-1 layers: . Truncating to the first layers gives , and the Eckart–Young theorem proves this is the best rank- approximation in Frobenius norm, with error .
Quick numeric example. Suppose has singular values . The full Frobenius norm is . Keeping only gives error — we captured 99.9% of the "signal" in just two layers. Keeping only gives error — still meaningful structure, but a lot is lost. The singular value spectrum tells you at a glance where to cut.
This fact powers image compression, recommender systems, LoRA fine-tuning, and noise filtering.
import numpy as np
rng = np.random.default_rng(0)
A = rng.normal(size=(60, 40)) @ rng.normal(size=(40, 40))
U, s, Vt = np.linalg.svd(A, full_matrices=False)
k = 10
A_k = U[:, :k] @ np.diag(s[:k]) @ Vt[:k, :]
err = np.linalg.norm(A - A_k, "fro")
tail = np.sqrt((s[k:] ** 2).sum())
print(f"rank-{k} error = {err:.4f}, tail = {tail:.4f}") # identical
Sweep the truncation rank below and watch the rank- reconstruction sharpen as the discarded singular values (and the Frobenius error ) shrink. A few layers capture almost all the signal.
Low-rank approximation: keep the biggest layers, drop the rest
A is rebuilt from its rank-1 layers σᵢ uᵢ vᵢᵀ, biggest singular value first. Truncating to the top k layers gives the provably best rank-k approximation (Eckart–Young), with Frobenius error √(Σ_{i>k} σᵢ²) — the height of the bars you discarded. The spectrum's steep drop then flat tail means a handful of layers carry almost all the signal.
Reading the spectrum
Plot the singular values in decreasing order. A steep drop followed by a flat noise floor means the matrix is effectively low-rank — the elbow tells you the true number of underlying factors.
Exercise
A 1000×800 matrix has singular values [95, 80, 60, 0.4, 0.3, 0.2, ...all below 0.4]. What's the best way to describe it?
Need a hint?
Where does the spectrum drop off a cliff?
Key takeaways
- SVD factors any matrix into rotation · stretch · rotation; singular values are the stretch factors, always real and ≥ 0
- , are eigenvectors of and ; , and PCA is the SVD of centered data
- : a stack of rank-1 layers, sorted by importance
- Truncating the stack gives the provably best rank-k approximation (Eckart–Young) — the engine behind compression, recommenders, denoising, and LoRA
Related concepts
- Eigenvalues & Eigenvectors: the square-symmetric special case SVD generalizes
- PCA: SVD applied to centered data
- PCA in Practice: choosing k from the spectrum, reconstruction error