Eigenvalues and Eigenvectors
When you apply a matrix to most vectors, both the direction and the length change. But there exist special vectors (eigenvectors) that only get scaled, never rotated. These special directions reveal the matrix's fundamental geometry — the geometry that decides what a transformation-based model (PCA, a spectral method, a deep network's stability) can and can't represent.
Real-world examples: PCA uses eigenvectors of the covariance matrix to find directions of maximum variance; Google PageRank is the dominant eigenvector of the web link graph; neural network stability analysis uses the spectral radius (largest absolute eigenvalue) to determine whether gradients will explode or vanish.
Eigenvectors: the directions a matrix only stretches
Grey = unit vectors, blue = where M sends them. Most directions rotate. Along the bold eigenvector lines, M·v stays on the line — it only scales by the eigenvalue λ. Symmetric matrices have perpendicular eigenvectors.
The eigenvalue equation
A non-zero vector is an eigenvector of if:
The scalar is the corresponding eigenvalue — it tells you how much the eigenvector stretches or shrinks.
- : stretches along this direction
- : shrinks along this direction
- : flips and scales
- : collapses — information is lost in this direction (rank deficiency)
Finding eigenvalues
Rewrite the equation: . For this to have a non-zero solution, the matrix must be singular:
This is the characteristic equation. For a matrix, it's a quadratic in .
For , the characteristic equation gives , . Substituting back into gives ; both rows reduce to , so the eigenvector is any scalar multiple of . Similarly yields . The set of vectors satisfying is called the null space of — finding it is always the last step. For the complete calculation, verification checks, and power iteration for large matrices:
Eigendecomposition
If has linearly independent eigenvectors, it can be diagonalized:
where and .
Symmetric matrices are special
For symmetric matrices (), eigenvalues are always real, eigenvectors are orthogonal, and where is orthogonal — a guarantee known as the spectral theorem. Covariance matrices are symmetric. This is why PCA always gives orthogonal components.
Connection to PCA
PCA of a dataset finds the eigenvectors of the covariance matrix .
- The eigenvector with the largest eigenvalue is the direction of maximum variance (first principal component).
- The eigenvector with the second largest eigenvalue is the next orthogonal direction of maximum variance.
- Projecting onto the top eigenvectors gives a rank- approximation that captures as much variance as possible.
# PCA via eigendecomposition
import numpy as np
rng = np.random.default_rng(0)
X = rng.standard_normal((100, 5)) # 100 samples, 5 features
X -= X.mean(axis=0) # center the data
C = (X.T @ X) / len(X) # covariance matrix
eigenvalues, V = np.linalg.eigh(C) # eigh for symmetric matrices
# Sort by descending eigenvalue
idx = np.argsort(eigenvalues)[::-1]
eigenvalues, V = eigenvalues[idx], V[:, idx]
explained_variance = eigenvalues / eigenvalues.sum()
print("Variance explained by PC1:", explained_variance[0])
X_2d = X @ V[:, :2] # project to 2D
Trace and determinant
Two scalar summaries of a matrix are directly tied to eigenvalues:
- Trace = sum of eigenvalues (for a covariance matrix, that sum is the total variance).
- Determinant = product of eigenvalues = signed volume scaling factor.
- means at least one eigenvalue is 0 → matrix is singular, not invertible.
Singular Value Decomposition (SVD)
Eigendecomposition is picky: it needs a square matrix, and its cleanest guarantees need symmetry. For any matrix, the singular value decomposition generalizes the idea:
with orthonormal and non-negative singular values on the diagonal of . For a square symmetric PSD matrix (like a covariance matrix), the two decompositions coincide: and .
Where , , come from, why truncating them gives the best low-rank approximation (Eckart–Young), and the compression applications are the next lesson.
Exercise
A matrix A has eigenvalue λ = 0. What does this imply?
Need a hint?
Av = 0·v = 0 for some non-zero v.
Exercise
If you scale an eigenvector v by 3 to get 3v, is 3v also an eigenvector?
Need a hint?
A(cv) = cAv for any scalar c.
Exercise
In PCA, the first principal component is the eigenvector of the covariance matrix with what property?
Need a hint?
PCA maximizes variance.
Common mistakes
- Assuming all matrices are diagonalizable: defective matrices (repeated eigenvalues with too few independent eigenvectors) cannot be diagonalized. The matrix has eigenvalue 0 with algebraic multiplicity 2 but only one eigenvector direction.
- Confusing eigenvalues with singular values: they coincide only for symmetric positive semi-definite matrices. For a general matrix, singular values are always non-negative, eigenvalues can be negative or complex, and the two sets are unrelated.
- Forgetting eigenvectors are defined only up to scalar multiple: only the direction of an eigenvector is meaningful, not its length or sign. Multiplying an eigenvector by any non-zero scalar gives another valid eigenvector for the same eigenvalue.
Key takeaways
- Eigenvectors are directions that a matrix only scales — the eigenvalue is the scale factor.
- Symmetric matrices have real, orthogonal eigenvectors — guaranteed clean decomposition.
- PCA = finding eigenvectors of the covariance matrix; eigenvalues = variance along each component.
- SVD generalizes to any matrix shape; singular values are always non-negative.
- — zero determinant means lost information (singular matrix).
Related concepts
- Matrices and Transformations: what eigendecomposition decomposes
- SVD & Low-Rank Approximation: the generalization to any matrix, continued next
- PCA & Dimensionality Reduction: eigendecomposition applied
- Calculus for ML: Derivatives: gradients point in the eigenvector direction at optima