Factor AnalysisOpen in Colab

Factor Analysis

Factor Analysis (FA) is a probabilistic latent variable model that explains a high-dimensional observation x∈Rnx \in \mathbb{R}^n as a linear combination of k≪nk \ll n hidden factors z∈Rkz \in \mathbb{R}^k plus independent noise.

The generative model

z∼N(0,Ik)z \sim \mathcal{N}(0, I_k) x∣z∼N(μ+Λz, Ψ)x \mid z \sim \mathcal{N}(\mu + \Lambda z,\, \Psi)
  • zz — the latent factor vector (kk-dimensional, typically k≪nk \ll n)
  • Λ∈Rn×k\Lambda \in \mathbb{R}^{n \times k} — the factor loading matrix; column jj is the "direction" of factor jj in observation space
  • μ∈Rn\mu \in \mathbb{R}^n — the mean of the observations
  • Ψ=diag(ψ1,…,ψn)\Psi = \text{diag}(\psi_1, \ldots, \psi_n) — a diagonal noise covariance; each observed dimension has its own independent noise variance

Marginalizing over zz gives the marginal:

x∼N(μ,  ΛΛ⊤+Ψ)x \sim \mathcal{N}(\mu,\; \Lambda\Lambda^\top + \Psi)

The covariance ΛΛ⊤+Ψ\Lambda\Lambda^\top + \Psi has low-rank structure (ΛΛ⊤\Lambda\Lambda^\top) plus diagonal noise (Ψ\Psi).

Intuition: cocktail party analogy

Imagine n=100n = 100 microphones recording k=3k = 3 speakers. Each microphone's reading xix_i is a linear combination of the three speakers' voices (the factors zjz_j) plus independent microphone noise (ψi\psi_i). Factor Analysis recovers the loading matrix Λ\Lambda that tells you how each microphone mixes the speakers, and the noise levels ψi\psi_i per microphone.

EM for Factor Analysis

Factor Analysis is fit via EM because the latent factors zz are unobserved.

E-step: Compute the posterior of zz given xx using the joint Gaussian:

z∣x∼N ⁣(μz∣x,  Σz∣x)z \mid x \sim \mathcal{N}\!\left(\mu_{z|x},\; \Sigma_{z|x}\right) Σz∣x=(I+Λ⊤Ψ−1Λ)−1,μz∣x=Σz∣x Λ⊤Ψ−1(x−μ)\Sigma_{z|x} = \left(I + \Lambda^\top \Psi^{-1} \Lambda\right)^{-1}, \qquad \mu_{z|x} = \Sigma_{z|x}\,\Lambda^\top \Psi^{-1}(x - \mu)

M-step: Re-estimate Λ\Lambda, μ\mu, and Ψ\Psi in closed form:

Λnew=(∑ix(i)E[z(i)]⊤)(∑iE[z(i)z(i)⊤])−1\Lambda_{\text{new}} = \left(\sum_i x^{(i)} E[z^{(i)}]^\top\right)\left(\sum_i E[z^{(i)}z^{(i)\top}]\right)^{-1} ψj=1m∑i(xj(i)−μj)2−Λj,: E[z(i)]\psi_j = \frac{1}{m}\sum_i \left(x_j^{(i)} - \mu_j\right)^2 - \Lambda_{j,:}\,E[z^{(i)}]

EM alternates these two steps until convergence.

Factor Analysis vs. PCA

PCAFactor Analysis
TypeDeterministic linear projectionProbabilistic latent variable model
Noise modelNone (or spherical: σ2I\sigma^2 I)Per-dimension: diagonal Ψ\Psi
ComponentsOrthogonal (from eigenvectors of covariance)Not necessarily orthogonal (can be rotated)
Rotation invarianceNo: PC directions are fixedYes: Λ\Lambda can be rotated without changing the marginal likelihood
Handles heteroscedastic noiseNoYes: each dimension has its own noise variance
Use caseDimensionality reduction, preprocessingIdentifying latent structure when dimensions have different noise levels

Key difference: PCA minimizes reconstruction error and implicitly assumes all dimensions have the same noise. Factor Analysis uses a per-dimension noise model Ψ\Psi, which makes it more appropriate when different measurements have different reliability (e.g., some sensors are noisy, others are precise).

Relationship to probabilistic PCA

Probabilistic PCA (PPCA) is a special case of Factor Analysis where Ψ=σ2I\Psi = \sigma^2 I (all dimensions have the same noise). In that limit:

  • The ML solution for Λ\Lambda recovers the top kk PCA directions (up to rotation)
  • PPCA gives a probabilistic interpretation to the PCA objective
  • Factor Analysis generalizes this by relaxing the equal-noise assumption

Identifiability and rotation

The Factor Analysis model is not identifiable without additional constraints. If RR is any orthogonal matrix, then ΛR\Lambda R and Λ\Lambda produce the same marginal distribution for xx (since (ΛR)(ΛR)⊤=ΛΛ⊤({\Lambda R})({\Lambda R})^\top = \Lambda\Lambda^\top). Common constraints:

  • Uniqueness factor analysis: constrain Λ⊤Ψ−1Λ\Lambda^\top\Psi^{-1}\Lambda to be diagonal
  • Varimax rotation: post-hoc rotation to maximize sparsity of the loading matrix
  • Confirmatory FA: impose prior knowledge about which factors affect which variables

When to use Factor Analysis

  • When you believe observations are generated from a small number of underlying latent causes
  • When different observed dimensions have different noise levels (heteroscedastic noise)
  • When you want a probabilistic model that allows uncertainty quantification on the latent factors
  • When you need to impute missing data (the EM E-step naturally handles missing observations)

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

RelatedReferenced by
Factor AnalysisIndependent Component Ana…Principal Component Analy…The EM Algorithm