Factor Analysis
Factor Analysis (FA) is a probabilistic latent variable model that explains a high-dimensional observation as a linear combination of hidden factors plus independent noise.
The generative model
- — the latent factor vector (-dimensional, typically )
- — the factor loading matrix; column is the "direction" of factor in observation space
- — the mean of the observations
- — a diagonal noise covariance; each observed dimension has its own independent noise variance
Marginalizing over gives the marginal:
The covariance has low-rank structure () plus diagonal noise ().
Intuition: cocktail party analogy
Imagine microphones recording speakers. Each microphone's reading is a linear combination of the three speakers' voices (the factors ) plus independent microphone noise (). Factor Analysis recovers the loading matrix that tells you how each microphone mixes the speakers, and the noise levels per microphone.
EM for Factor Analysis
Factor Analysis is fit via EM because the latent factors are unobserved.
E-step: Compute the posterior of given using the joint Gaussian:
M-step: Re-estimate , , and in closed form:
EM alternates these two steps until convergence.
Factor Analysis vs. PCA
| PCA | Factor Analysis | |
|---|---|---|
| Type | Deterministic linear projection | Probabilistic latent variable model |
| Noise model | None (or spherical: ) | Per-dimension: diagonal |
| Components | Orthogonal (from eigenvectors of covariance) | Not necessarily orthogonal (can be rotated) |
| Rotation invariance | No: PC directions are fixed | Yes: can be rotated without changing the marginal likelihood |
| Handles heteroscedastic noise | No | Yes: each dimension has its own noise variance |
| Use case | Dimensionality reduction, preprocessing | Identifying latent structure when dimensions have different noise levels |
Key difference: PCA minimizes reconstruction error and implicitly assumes all dimensions have the same noise. Factor Analysis uses a per-dimension noise model , which makes it more appropriate when different measurements have different reliability (e.g., some sensors are noisy, others are precise).
Relationship to probabilistic PCA
Probabilistic PCA (PPCA) is a special case of Factor Analysis where (all dimensions have the same noise). In that limit:
- The ML solution for recovers the top PCA directions (up to rotation)
- PPCA gives a probabilistic interpretation to the PCA objective
- Factor Analysis generalizes this by relaxing the equal-noise assumption
Identifiability and rotation
The Factor Analysis model is not identifiable without additional constraints. If is any orthogonal matrix, then and produce the same marginal distribution for (since ). Common constraints:
- Uniqueness factor analysis: constrain to be diagonal
- Varimax rotation: post-hoc rotation to maximize sparsity of the loading matrix
- Confirmatory FA: impose prior knowledge about which factors affect which variables
When to use Factor Analysis
- When you believe observations are generated from a small number of underlying latent causes
- When different observed dimensions have different noise levels (heteroscedastic noise)
- When you want a probabilistic model that allows uncertainty quantification on the latent factors
- When you need to impute missing data (the EM E-step naturally handles missing observations)
Related concepts
- The EM Algorithm: the coordinate-ascent procedure that fits FA's latent factors
- Principal Component Analysis: the deterministic cousin; PPCA is FA with isotropic noise
- Independent Component Analysis: a latent-factor model that instead assumes non-Gaussian, independent sources