Eigenvalues and EigenvectorsOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Eigenvalues and Eigenvectors

When you apply a matrix A\mathbf{A} to most vectors, both the direction and the length change. But there exist special vectors (eigenvectors) that only get scaled, never rotated. These special directions reveal the matrix's fundamental geometry — the geometry that decides what a transformation-based model (PCA, a spectral method, a deep network's stability) can and can't represent.

Real-world examples: PCA uses eigenvectors of the covariance matrix to find directions of maximum variance; Google PageRank is the dominant eigenvector of the web link graph; neural network stability analysis uses the spectral radius (largest absolute eigenvalue) to determine whether gradients will explode or vanish.

Eigenvectors: the directions a matrix only stretches

λ₁2.44
λ₂0.56

Grey = unit vectors, blue = where M sends them. Most directions rotate. Along the bold eigenvector lines, M·v stays on the line — it only scales by the eigenvalue λ. Symmetric matrices have perpendicular eigenvectors.

The eigenvalue equation

A non-zero vector v\mathbf{v} is an eigenvector of A\mathbf{A} if:

Av=λv\mathbf{A}\mathbf{v} = \lambda\mathbf{v}

The scalar λ\lambda is the corresponding eigenvalue — it tells you how much the eigenvector stretches or shrinks.

  • λ>1\lambda > 1: stretches along this direction
  • 0<λ<10 < \lambda < 1: shrinks along this direction
  • λ<0\lambda < 0: flips and scales
  • λ=0\lambda = 0: collapses — information is lost in this direction (rank deficiency)

Finding eigenvalues

Rewrite the equation: (A−λI)v=0(\mathbf{A} - \lambda \mathbf{I})\mathbf{v} = \mathbf{0}. For this to have a non-zero solution, the matrix must be singular:

det⁡(A−λI)=0\det(\mathbf{A} - \lambda \mathbf{I}) = 0

This is the characteristic equation. For a 2×22 \times 2 matrix, it's a quadratic in λ\lambda.

For A=[4123]\mathbf{A} = \begin{bmatrix}4&1\\2&3\end{bmatrix}, the characteristic equation λ2−7λ+10=0\lambda^2 - 7\lambda + 10 = 0 gives λ1=5\lambda_1 = 5, λ2=2\lambda_2 = 2. Substituting λ1=5\lambda_1 = 5 back into (A−5I)v=0(\mathbf{A}-5\mathbf{I})\mathbf{v}=\mathbf{0} gives [−112−2]v=0\begin{bmatrix}-1&1\\2&-2\end{bmatrix}\mathbf{v}=\mathbf{0}; both rows reduce to v1=v2v_1 = v_2, so the eigenvector is any scalar multiple of [1,1]⊤[1,1]^\top. Similarly λ2=2\lambda_2 = 2 yields [1,−2]⊤[1,-2]^\top. The set of vectors satisfying (A−λI)v=0(A-\lambda I)\mathbf{v}=\mathbf{0} is called the null space of A−λIA-\lambda I — finding it is always the last step. For the complete calculation, verification checks, and power iteration for large matrices:

Deep dive
Computing Eigenvalues & Eigenvectors
Includes runnable notebook

Eigendecomposition

If A\mathbf{A} has nn linearly independent eigenvectors, it can be diagonalized:

A=VΛV−1\mathbf{A} = \mathbf{V}\boldsymbol{\Lambda}\mathbf{V}^{-1}

where V=[v1∣⋯∣vn]\mathbf{V} = [\mathbf{v}_1 | \cdots | \mathbf{v}_n] and Λ=diag(λ1,…,λn)\boldsymbol{\Lambda} = \text{diag}(\lambda_1, \ldots, \lambda_n).

Symmetric matrices are special

For symmetric matrices (A=A⊤\mathbf{A} = \mathbf{A}^\top), eigenvalues are always real, eigenvectors are orthogonal, and A=QΛQ⊤\mathbf{A} = \mathbf{Q}\boldsymbol{\Lambda}\mathbf{Q}^\top where Q\mathbf{Q} is orthogonal — a guarantee known as the spectral theorem. Covariance matrices are symmetric. This is why PCA always gives orthogonal components.

Connection to PCA

PCA of a dataset X\mathbf{X} finds the eigenvectors of the covariance matrix C=1nX⊤X\mathbf{C} = \frac{1}{n}\mathbf{X}^\top\mathbf{X}.

  • The eigenvector with the largest eigenvalue is the direction of maximum variance (first principal component).
  • The eigenvector with the second largest eigenvalue is the next orthogonal direction of maximum variance.
  • Projecting X\mathbf{X} onto the top kk eigenvectors gives a rank-kk approximation that captures as much variance as possible.
XPCA=XVk,Vk=[v1∣⋯∣vk]\mathbf{X}_\text{PCA} = \mathbf{X}\mathbf{V}_k, \quad \mathbf{V}_k = [\mathbf{v}_1 | \cdots | \mathbf{v}_k]
Python
# PCA via eigendecomposition
import numpy as np

rng = np.random.default_rng(0)
X = rng.standard_normal((100, 5))  # 100 samples, 5 features
X -= X.mean(axis=0)               # center the data

C = (X.T @ X) / len(X)           # covariance matrix
eigenvalues, V = np.linalg.eigh(C) # eigh for symmetric matrices

# Sort by descending eigenvalue
idx = np.argsort(eigenvalues)[::-1]
eigenvalues, V = eigenvalues[idx], V[:, idx]

explained_variance = eigenvalues / eigenvalues.sum()
print("Variance explained by PC1:", explained_variance[0])

X_2d = X @ V[:, :2]              # project to 2D

Trace and determinant

Two scalar summaries of a matrix are directly tied to eigenvalues:

tr(A)=∑iAii=∑iλi,det⁡(A)=∏iλi\text{tr}(\mathbf{A}) = \sum_i A_{ii} = \sum_i \lambda_i, \qquad \det(\mathbf{A}) = \prod_i \lambda_i
  • Trace = sum of eigenvalues (for a covariance matrix, that sum is the total variance).
  • Determinant = product of eigenvalues = signed volume scaling factor.
    • det⁡=0\det = 0 means at least one eigenvalue is 0 → matrix is singular, not invertible.

Singular Value Decomposition (SVD)

Eigendecomposition is picky: it needs a square matrix, and its cleanest guarantees need symmetry. For any matrix, the singular value decomposition generalizes the idea:

A=UΣV⊤\mathbf{A} = \mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^\top

with orthonormal U,V\mathbf{U}, \mathbf{V} and non-negative singular values σi\sigma_i on the diagonal of Σ\boldsymbol{\Sigma}. For a square symmetric PSD matrix (like a covariance matrix), the two decompositions coincide: σi=λi\sigma_i = \lambda_i and U=V=Q\mathbf{U} = \mathbf{V} = \mathbf{Q}.

Where U\mathbf{U}, Σ\boldsymbol{\Sigma}, V\mathbf{V} come from, why truncating them gives the best low-rank approximation (Eckart–Young), and the compression applications are the next lesson.

Exercise

A matrix A has eigenvalue λ = 0. What does this imply?

Need a hint?

Av = 0·v = 0 for some non-zero v.

Exercise

If you scale an eigenvector v by 3 to get 3v, is 3v also an eigenvector?

Need a hint?

A(cv) = cAv for any scalar c.

Exercise

In PCA, the first principal component is the eigenvector of the covariance matrix with what property?

Need a hint?

PCA maximizes variance.

Common mistakes

  1. Assuming all matrices are diagonalizable: defective matrices (repeated eigenvalues with too few independent eigenvectors) cannot be diagonalized. The 2×22\times2 matrix [0100]\begin{bmatrix}0&1\\0&0\end{bmatrix} has eigenvalue 0 with algebraic multiplicity 2 but only one eigenvector direction.
  2. Confusing eigenvalues with singular values: they coincide only for symmetric positive semi-definite matrices. For a general matrix, singular values are always non-negative, eigenvalues can be negative or complex, and the two sets are unrelated.
  3. Forgetting eigenvectors are defined only up to scalar multiple: only the direction of an eigenvector is meaningful, not its length or sign. Multiplying an eigenvector by any non-zero scalar gives another valid eigenvector for the same eigenvalue.

Key takeaways

  • Eigenvectors are directions that a matrix only scales — the eigenvalue is the scale factor.
  • Symmetric matrices have real, orthogonal eigenvectors — guaranteed clean decomposition.
  • PCA = finding eigenvectors of the covariance matrix; eigenvalues = variance along each component.
  • SVD generalizes to any matrix shape; singular values are always non-negative.
  • det⁡(A)=∏λi\det(\mathbf{A}) = \prod \lambda_i — zero determinant means lost information (singular matrix).

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

RelatedGo deeper
Eigenvalues and EigenvectorsVectors and SpacesMatrices and Transformati…SVD & Low-Rank Approximat…Principal Component Analy…Derivatives and GradientsMultivariable OptimizationComputing Eigenvalues & E…Newton's Method for Optim…Vanishing & Exploding Gra…