Independent Component Analysis (ICA)Open in Colab

Independent Component Analysis (ICA)

ICA recovers statistically independent source signals from mixed observations — without knowing the mixing process. The canonical example is the cocktail party problem: nn microphones record nn speakers talking simultaneously; ICA unmixes the signals.

The model

Assume nn independent source signals s1,…,sns_1, \ldots, s_n are mixed by an unknown matrix A∈Rn×nA \in \mathbb{R}^{n \times n}:

x=Asx = A s

We observe xx but not ss or AA. The goal: find the unmixing matrix W=A−1W = A^{-1} such that s=Wxs = Wx.

Why Gaussianity fails (and non-Gaussianity saves us)

The Central Limit Theorem says that a sum of independent random variables tends toward Gaussian, regardless of the source distributions. Therefore, mixtures of independent sources are more Gaussian than the sources themselves. This gives us a recovery strategy:

Find the unmixing WW that makes the recovered signals s^=Wx\hat{s} = Wx as non-Gaussian as possible.

Maximally non-Gaussian → maximally unmixed.

The one exception: if the sources are themselves Gaussian, ICA fails — Gaussian independence cannot be distinguished from Gaussian mixture in this way. (This is one key difference from Factor Analysis, which assumes Gaussian latent variables.)

Measures of non-Gaussianity

Kurtosis: kurt(s)=E[s4]−3(E[s2])2\text{kurt}(s) = E[s^4] - 3(E[s^2])^2. Kurtosis is 0 for a Gaussian; positive for heavy-tailed distributions (super-Gaussian), negative for light-tailed (sub-Gaussian). ICA maximizes absolute kurtosis.

Negentropy: J(s)=H(sGaussian)−H(s)J(s) = H(s_\text{Gaussian}) - H(s), where HH is differential entropy and sGaussians_\text{Gaussian} has the same variance. Gaussian has maximum entropy, so J(s)≥0J(s) \ge 0 with equality iff ss is Gaussian. Maximizing negentropy = minimizing entropy = maximizing non-Gaussianity.

FastICA approximates negentropy using a smooth nonlinearity GG:

J(s)≈[E[G(s)]−E[G(ν)]]2J(s) \approx \left[E[G(s)] - E[G(\nu)]\right]^2

where ν∼N(0,1)\nu \sim \mathcal{N}(0,1). Common choices: G(u)=log⁡cosh⁡(u)G(u) = \log \cosh(u) (general) or G(u)=−e−u2/2G(u) = -e^{-u^2/2} (sparse sources).

Watch FastICA run

FastICA — and what happens when the sources are Gaussian

1/6

Two independent sources — a sawtooth and a square wave — reach two microphones through an unknown mixing matrix. The mixtures are what you get; the sources are what you want. The kurtosis column is the whole strategy in one number: each **source** has excess kurtosis -1.19 and -2.00, far from Gaussian, while the **mixtures** sit at -0.77 and -1.54 — measurably closer to zero. Adding independent things Gaussianises them, exactly as the central limit theorem promises, so *un*mixing means searching for the most non-Gaussian direction.

python
1X = X - X.mean(axis=1, keepdims=True)
2
3# whiten: decorrelate, unit variance
4E, D = eig(cov(X))
5X = D**-0.5 @ E.T @ X
6
7w = random_unit_vector()
8for it in range(max_iter):
9 s = w @ X
10
11 # fixed point of the negentropy
12 # approximation, G = log cosh
13 g, gp = tanh(s), 1 - tanh(s)**2
14 w_new = (X * g).mean(1) - gp.mean() * w
15
16 w_new /= norm(w_new)
17 if abs(abs(w_new @ w) - 1) < tol:
18 break
19 w = w_new
state
the two mixtures — a sheared, correlated cloud
-333-3channel 1channel 2
excess kurtosis (0 = Gaussian)
signalkurtosis
source 1 (sawtooth)-1.192
source 2 (square)-2.000
mixture 1-0.772
mixture 2-1.537

Two sources through an unknown 2x2 mixer. The kurtosis table shows the mixtures measurably closer to Gaussian than the sources, which is the whole strategy; whitening then reduces unmixing to a search over one rotation angle. Watch the fixed-point iteration swing the direction while the projection's kurtosis climbs. Then two measured payoffs: with Gaussian sources the identical pipeline converges confidently to a different arbitrary direction on every run, because there is no maximum to find; and across 12 runs the recovered signals match the true sources every time while their sign and order wander freely, which is exactly what the mixing model leaves undetermined.

What to notice:

  • Frame 1: the strategy in one column. The sources have excess kurtosis −1.19 and −2.00; the mixtures sit at −0.77 and −1.54, measurably closer to Gaussian. That gap is the entire signal FastICA exploits.
  • Frame 2: whitening does not solve it. Covariance goes from an off-diagonal of 0.79 to 0.0000, and the data is still mixed. Whitening removes second-order dependence only, and independence is a statement about all orders. What it buys is that the remaining unmixing is a pure rotation, so the search is over one angle.
  • Frame 3: cubic convergence. From 85° the direction reaches 38.1° in four iterations while the projection's |kurtosis| climbs 0.758 → 2.000, and the update size falls 4.6e-2 → 1.5e-1 → 6.2e-4 → 8.5e-14.
  • Frame 5: the payoff. With Gaussian sources the identical pipeline converges confidently to a different arbitrary direction every run — spread 82.7° of an available 90°, against 0.36° for non-Gaussian sources. It does not error or fail to converge, which is what makes the failure dangerous: there is simply no maximum to find, so the answer is wherever the search started.
  • Frame 6: the indeterminacies cost only bookkeeping. Across 12 runs the components match the true sources with |correlation| 1.000 every time, while the output order and signs vary. Match components by correlation, never by index.

One measurement note carried in the trace: separation quality there is reported as cross-talk (how much of the other source leaks in), not as match quality. A recovered basis is some rotation of the true one, so the matched correlation is ∣cos⁡θ∣|\cos\theta| and cannot fall below 0.71 however arbitrary θ\theta is — a completely random direction scores ~0.92 and looks like success.

Bell-Sejnowski algorithm (MLE ICA)

Model each source sis_i as having a known, non-Gaussian density psip_{s_i}. The log-likelihood of WW given observations x(1),…,x(m)x^{(1)}, \ldots, x^{(m)}:

ℓ(W)=∑i=1m∑j=1nlog⁡psj(wj⊤x(i))+mlog⁡∣W∣\ell(W) = \sum_{i=1}^{m}\sum_{j=1}^{n} \log p_{s_j}(w_j^\top x^{(i)}) + m \log |W|

where wj⊤w_j^\top is the jj-th row of WW. The gradient ascent update:

W←W+α[(1−2g(w1⊤x)⋮1−2g(wn⊤x))x⊤+(W⊤)−1]W \leftarrow W + \alpha \left[\begin{pmatrix} 1 - 2g(w_1^\top x) \\ \vdots \\ 1 - 2g(w_n^\top x) \end{pmatrix} x^\top + (W^\top)^{-1}\right]

where g=ps′/psg = p_s'/p_s is the score function of the source density. For a standard sigmoid-shaped density, g(⋅)=σ(⋅)g(\cdot) = \sigma(\cdot) and 1−2g(u)=tanh⁡(u/2)1 - 2g(u) = \tanh(u/2).

The log⁡∣W∣\log|W| term (from the change-of-variables Jacobian) is crucial — it prevents the trivial solution W=0W = 0.

ICA vs. PCA

PCAICA
GoalDecorrelate (find orthogonal directions of maximum variance)Find statistically independent components
Second-orderYes: only uses covariance matrixNo: requires higher-order statistics
RotationFixed (eigenvectors)Rotation is the parameter: can be anything
GaussianityOptimal for Gaussian dataFails for Gaussian sources
ApplicationsCompression, preprocessingBlind source separation, neural decoding

Key insight: PCA removes linear correlations; ICA removes all statistical dependencies. Two variables can be uncorrelated but not independent (e.g., X∼U[−1,1]X \sim \mathcal{U}[-1,1], Y=X2Y = X^2: E[XY]=0E[XY] = 0 but YY is determined by XX). ICA removes even these higher-order dependencies.

Practical applications

  • Audio source separation: unmix nn speakers from nn microphones
  • EEG/MEG: separate brain signal components (artifacts like eye blinks from neural oscillations)
  • Financial data: find independent risk factors driving correlated stock returns
  • fMRI: identify independent spatial modes of brain activity

Limitations

  1. Requires nn sources = nn observations (square mixing matrix). Extensions exist for overcomplete ICA (more sources than observed mixtures).
  2. Order and scale are ambiguous: WA=IWA = I but (WD)(D−1A)(WD)(D^{-1}A) is also a valid solution for any diagonal DD. Convention: normalize sources to unit variance.
  3. Fails for Gaussian sources: cannot distinguish independent Gaussians from their rotations.
  4. Sensitive to outliers: kurtosis maximization is particularly sensitive to heavy tails.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

RelatedReferenced by
Independent Component Analysis (I…Factor AnalysisPrincipal Component Analy…The EM Algorithm