Independent Component Analysis (ICA)
ICA recovers statistically independent source signals from mixed observations — without knowing the mixing process. The canonical example is the cocktail party problem: microphones record speakers talking simultaneously; ICA unmixes the signals.
The model
Assume independent source signals are mixed by an unknown matrix :
We observe but not or . The goal: find the unmixing matrix such that .
Why Gaussianity fails (and non-Gaussianity saves us)
The Central Limit Theorem says that a sum of independent random variables tends toward Gaussian, regardless of the source distributions. Therefore, mixtures of independent sources are more Gaussian than the sources themselves. This gives us a recovery strategy:
Find the unmixing that makes the recovered signals as non-Gaussian as possible.
Maximally non-Gaussian → maximally unmixed.
The one exception: if the sources are themselves Gaussian, ICA fails — Gaussian independence cannot be distinguished from Gaussian mixture in this way. (This is one key difference from Factor Analysis, which assumes Gaussian latent variables.)
Measures of non-Gaussianity
Kurtosis: . Kurtosis is 0 for a Gaussian; positive for heavy-tailed distributions (super-Gaussian), negative for light-tailed (sub-Gaussian). ICA maximizes absolute kurtosis.
Negentropy: , where is differential entropy and has the same variance. Gaussian has maximum entropy, so with equality iff is Gaussian. Maximizing negentropy = minimizing entropy = maximizing non-Gaussianity.
FastICA approximates negentropy using a smooth nonlinearity :
where . Common choices: (general) or (sparse sources).
Watch FastICA run
FastICA — and what happens when the sources are Gaussian
Two independent sources — a sawtooth and a square wave — reach two microphones through an unknown mixing matrix. The mixtures are what you get; the sources are what you want. The kurtosis column is the whole strategy in one number: each **source** has excess kurtosis -1.19 and -2.00, far from Gaussian, while the **mixtures** sit at -0.77 and -1.54 — measurably closer to zero. Adding independent things Gaussianises them, exactly as the central limit theorem promises, so *un*mixing means searching for the most non-Gaussian direction.
| signal | kurtosis |
|---|---|
| source 1 (sawtooth) | -1.192 |
| source 2 (square) | -2.000 |
| mixture 1 | -0.772 |
| mixture 2 | -1.537 |
Two sources through an unknown 2x2 mixer. The kurtosis table shows the mixtures measurably closer to Gaussian than the sources, which is the whole strategy; whitening then reduces unmixing to a search over one rotation angle. Watch the fixed-point iteration swing the direction while the projection's kurtosis climbs. Then two measured payoffs: with Gaussian sources the identical pipeline converges confidently to a different arbitrary direction on every run, because there is no maximum to find; and across 12 runs the recovered signals match the true sources every time while their sign and order wander freely, which is exactly what the mixing model leaves undetermined.
What to notice:
- Frame 1: the strategy in one column. The sources have excess kurtosis −1.19 and −2.00; the mixtures sit at −0.77 and −1.54, measurably closer to Gaussian. That gap is the entire signal FastICA exploits.
- Frame 2: whitening does not solve it. Covariance goes from an off-diagonal of 0.79 to 0.0000, and the data is still mixed. Whitening removes second-order dependence only, and independence is a statement about all orders. What it buys is that the remaining unmixing is a pure rotation, so the search is over one angle.
- Frame 3: cubic convergence. From 85° the direction reaches 38.1° in four iterations while the projection's |kurtosis| climbs 0.758 → 2.000, and the update size falls 4.6e-2 → 1.5e-1 → 6.2e-4 → 8.5e-14.
- Frame 5: the payoff. With Gaussian sources the identical pipeline converges confidently to a different arbitrary direction every run — spread 82.7° of an available 90°, against 0.36° for non-Gaussian sources. It does not error or fail to converge, which is what makes the failure dangerous: there is simply no maximum to find, so the answer is wherever the search started.
- Frame 6: the indeterminacies cost only bookkeeping. Across 12 runs the components match the true sources with |correlation| 1.000 every time, while the output order and signs vary. Match components by correlation, never by index.
One measurement note carried in the trace: separation quality there is reported as cross-talk (how much of the other source leaks in), not as match quality. A recovered basis is some rotation of the true one, so the matched correlation is and cannot fall below 0.71 however arbitrary is — a completely random direction scores ~0.92 and looks like success.
Bell-Sejnowski algorithm (MLE ICA)
Model each source as having a known, non-Gaussian density . The log-likelihood of given observations :
where is the -th row of . The gradient ascent update:
where is the score function of the source density. For a standard sigmoid-shaped density, and .
The term (from the change-of-variables Jacobian) is crucial — it prevents the trivial solution .
ICA vs. PCA
| PCA | ICA | |
|---|---|---|
| Goal | Decorrelate (find orthogonal directions of maximum variance) | Find statistically independent components |
| Second-order | Yes: only uses covariance matrix | No: requires higher-order statistics |
| Rotation | Fixed (eigenvectors) | Rotation is the parameter: can be anything |
| Gaussianity | Optimal for Gaussian data | Fails for Gaussian sources |
| Applications | Compression, preprocessing | Blind source separation, neural decoding |
Key insight: PCA removes linear correlations; ICA removes all statistical dependencies. Two variables can be uncorrelated but not independent (e.g., , : but is determined by ). ICA removes even these higher-order dependencies.
Practical applications
- Audio source separation: unmix speakers from microphones
- EEG/MEG: separate brain signal components (artifacts like eye blinks from neural oscillations)
- Financial data: find independent risk factors driving correlated stock returns
- fMRI: identify independent spatial modes of brain activity
Limitations
- Requires sources = observations (square mixing matrix). Extensions exist for overcomplete ICA (more sources than observed mixtures).
- Order and scale are ambiguous: but is also a valid solution for any diagonal . Convention: normalize sources to unit variance.
- Fails for Gaussian sources: cannot distinguish independent Gaussians from their rotations.
- Sensitive to outliers: kurtosis maximization is particularly sensitive to heavy tails.
Related concepts
- Principal Component Analysis: decorrelation (second-order) vs ICA's full statistical independence
- The EM Algorithm: the latent-variable fitting framework behind probabilistic source models
- Factor Analysis: the Gaussian-latent counterpart ICA deliberately departs from