Gaussian Mixture ModelsOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Gaussian Mixture Models (GMMs)

K-Means assigns each point to exactly one cluster. GMMs assign soft probabilities — each point belongs to every cluster with some probability. This captures uncertainty and handles elliptical clusters. The shift from K-Means to a mixture of Gaussians is a hypothesis-space change: a richer, probabilistic model family for the same data, and the next lesson's EM is the optimization that fits it.

Deep dive
The ML Project Loop
Includes runnable notebook

Real-world examples: Speaker identification, image segmentation, density estimation, anomaly detection, semi-supervised learning.

The model

A GMM assumes data is generated from a mixture of KK Gaussian distributions:

P(x)=∑k=1Kπk⋅N(x∣μk,Σk)P(\mathbf{x}) = \sum_{k=1}^{K} \pi_k \cdot \mathcal{N}(\mathbf{x} \mid \boldsymbol{\mu}_k, \boldsymbol{\Sigma}_k)
  • πk\pi_k: mixing weight (probability of component kk), ∑πk=1\sum \pi_k = 1
  • μk\boldsymbol{\mu}_k: mean of component kk
  • Σk\boldsymbol{\Sigma}_k: covariance of component kk

Intuition

Imagine data comes from three different cameras with different lighting and angles. Each camera produces a Gaussian cloud of points. The GMM figures out which camera took each photo and the settings (mean, spread) of each camera.

Posterior probability (responsibility)

Given a data point xi\mathbf{x}_i, the probability it came from component kk is:

γik=P(zi=k∣xi)=πk⋅N(xi∣μk,Σk)∑j=1Kπj⋅N(xi∣μj,Σj)\gamma_{ik} = P(z_i = k \mid \mathbf{x}_i) = \frac{\pi_k \cdot \mathcal{N}(\mathbf{x}_i \mid \boldsymbol{\mu}_k, \boldsymbol{\Sigma}_k)}{\sum_{j=1}^{K} \pi_j \cdot \mathcal{N}(\mathbf{x}_i \mid \boldsymbol{\mu}_j, \boldsymbol{\Sigma}_j)}

This is Bayes' theorem — the responsibility of component kk for point ii.

EM for Gaussian Mixture Models

π=0.33π=0.33π=0.33feature x₁feature x₂
iteration0
log-likelihood-160.8
phaseinit
Component 1
Component 2
Component 3

Each EM step updates the component means and covariances (M-step) then re-computes soft responsibilities (E-step). Point colors blend between the three component colors weighted by responsibility. Ellipses show the 2σ contour of each Gaussian. Log-likelihood increases monotonically.

Worked example (by hand)

Two 1D components, equal weights π1=π2=0.5\pi_1=\pi_2=0.5, means μ1=0, μ2=5\mu_1=0,\,\mu_2=5, both σ=1\sigma=1. What is the responsibility of each for the point x=2x=2?

Plug into the 1D Gaussian N(x∣μ,σ)=12πe−(x−μ)2/2σ2\mathcal N(x\mid\mu,\sigma)=\tfrac{1}{\sqrt{2\pi}}e^{-(x-\mu)^2/2\sigma^2} (the 12π\tfrac{1}{\sqrt{2\pi}} cancels in the ratio, but we keep it for clarity):

N(2∣0,1)=12πe−2=0.0540,N(2∣5,1)=12πe−4.5=0.00443.\mathcal N(2\mid 0,1) = \tfrac{1}{\sqrt{2\pi}}e^{-2} = 0.0540, \qquad \mathcal N(2\mid 5,1) = \tfrac{1}{\sqrt{2\pi}}e^{-4.5} = 0.00443.

Weight by πk\pi_k (both 0.50.5) and normalize:

γ1=0.5 (0.0540)0.5 (0.0540)+0.5 (0.00443)=0.05400.0584=0.924,γ2=0.076.\gamma_1 = \frac{0.5\,(0.0540)}{0.5\,(0.0540) + 0.5\,(0.00443)} = \frac{0.0540}{0.0584} = 0.924, \qquad \gamma_2 = 0.076.

So x=2x=2 is 92.4% component 1, 7.6% component 2 — a soft assignment. K-Means would hard-assign it 100% to component 1 and throw away that 7.6% of doubt. Points exactly between the means (x=2.5x=2.5) split 50/5050/50.

Python
import numpy as np
from scipy.stats import multivariate_normal

def compute_responsibilities(X, means, covs, weights):
    K = len(weights)
    responsibilities = np.zeros((len(X), K))
    for k in range(K):
        responsibilities[:, k] = weights[k] * multivariate_normal.pdf(X, means[k], covs[k])
    responsibilities /= responsibilities.sum(axis=1, keepdims=True)
    return responsibilities

# Example: 2 components
X = np.array([[1,2], [1.5,2.5], [8,9], [8.5,9.5]])
means = [np.array([1.25, 2.25]), np.array([8.25, 9.25])]
covs = [np.eye(2)*0.1, np.eye(2)*0.1]
weights = [0.5, 0.5]

gamma = compute_responsibilities(X, means, covs, weights)
print("Responsibilities:")
print(gamma.round(3))
# First two points → component 0, last two → component 1

GMM vs K-Means

PropertyK-MeansGMM
Cluster shapeSphericalElliptical
AssignmentHard (0 or 1)Soft (probability)
Parametersμk\boldsymbol{\mu}_kμk,Σk,πk\boldsymbol{\mu}_k, \boldsymbol{\Sigma}_k, \pi_k
OptimizationAlternating assign/updateEM algorithm
Model selectionElbow/silhouetteBIC/AIC

K-Means is a special case of GMM where all covariances are σ2I\sigma^2 I (spherical) and responsibilities are hard-assigned.

Maximum Likelihood Estimation

Parameters θ={πk,μk,Σk}\theta = \{\pi_k, \boldsymbol{\mu}_k, \boldsymbol{\Sigma}_k\} are found by maximizing the log-likelihood:

log⁡P(X∣θ)=∑i=1nlog⁡(∑k=1Kπk⋅N(xi∣μk,Σk))\log P(X \mid \theta) = \sum_{i=1}^{n} \log \left( \sum_{k=1}^{K} \pi_k \cdot \mathcal{N}(\mathbf{x}_i \mid \boldsymbol{\mu}_k, \boldsymbol{\Sigma}_k) \right)

Because of the log-sum (a log of a sum can't be split), there's no closed-form solution. The EM algorithm (next lesson) gets around this by alternating: compute responsibilities γik\gamma_{ik} (E-step), then update parameters as responsibility-weighted statistics (M-step):

Nk=∑iγik,μk=1Nk∑iγikxi,Σk=1Nk∑iγik(xi−μk)(xi−μk)⊤,πk=Nkn.N_k = \sum_i \gamma_{ik},\quad \boldsymbol{\mu}_k = \frac{1}{N_k}\sum_i \gamma_{ik}\mathbf{x}_i,\quad \boldsymbol{\Sigma}_k = \frac{1}{N_k}\sum_i \gamma_{ik}(\mathbf{x}_i-\boldsymbol{\mu}_k)(\mathbf{x}_i-\boldsymbol{\mu}_k)^\top,\quad \pi_k = \frac{N_k}{n}.

These are exactly the K-Means update equations with the hard 0/1 membership replaced by the soft γik\gamma_{ik}, which is why a GMM is the soft generalization of K-Means.

Covariance types

scikit-learn supports four covariance structures:

TypeParameters per clusterShape
fulld(d+3)/2d(d+3)/2Arbitrary ellipsoids
tieddd (mean only) + one shared d(d+1)2\frac{d(d+1)}{2} covarianceSame ellipse for all
diag2d2dAxis-aligned ellipses
sphericald+1d+1Circles

(tied shares a single covariance matrix across all KK clusters, so each cluster only adds its own mean — strictly fewer parameters than full, which is the point of using it.)

Python
from sklearn.mixture import GaussianMixture

# Fit GMM with full covariance
gmm = GaussianMixture(n_components=3, covariance_type='full', random_state=42)
gmm.fit(X)

print(f"Means:\n{gmm.means_}")
print(f"Weights: {gmm.weights_}")
print(f"Log-likelihood: {gmm.score(X) * len(X):.2f}")

# Predict responsibilities
probs = gmm.predict_proba(X)
labels = gmm.predict(X)

Choosing the number of components

BIC (Bayesian Information Criterion)

BIC=−2log⁡P(X∣θ^)+kln⁡(n)\text{BIC} = -2 \log P(X \mid \hat{\theta}) + k \ln(n)

where kk is the total number of free parameters in the model (not the number of components KK) and nn is the number of data points. Penalizes complexity more than AIC — prefers simpler models.

AIC (Akaike Information Criterion)

AIC=−2log⁡P(X∣θ^)+2k\text{AIC} = -2 \log P(X \mid \hat{\theta}) + 2k

Less penalty than BIC — may select more complex models.

Python
import matplotlib.pyplot as plt

# Enough data to actually compare 1-7 components
rng = np.random.default_rng(42)
X = np.vstack([rng.normal([0, 0], 1.0, (150, 2)),
               rng.normal([5, 5], 1.5, (150, 2)),
               rng.normal([0, 6], 0.8, (150, 2))])   # 3 true clusters

n_components_range = range(1, 8)
bics = []
aics = []

for n in n_components_range:
    gmm = GaussianMixture(n_components=n, random_state=42)
    gmm.fit(X)
    bics.append(gmm.bic(X))
    aics.append(gmm.aic(X))
# Both criteria bottom out at n=3 — the true number of clusters

plt.figure(figsize=(8, 5))
plt.plot(n_components_range, bics, 'bo-', label='BIC')
plt.plot(n_components_range, aics, 'rs-', label='AIC')
plt.xlabel('Number of components')
plt.ylabel('Information criterion')
plt.legend()
plt.title('Choosing K with BIC/AIC')
plt.show()

BIC vs AIC

BIC penalizes complexity more heavily (grows with ln⁡n\ln n). Use BIC when you want a parsimonious model. Use AIC when you suspect the true model is complex. For small datasets, BIC is safer.

Exercise

In a GMM, what does the responsibility γ_ik represent?

Need a hint?

It's a probability given the data and current parameters.

Exercise

In scikit-learn's GMM, what does covariance_type='diag' assume?

Need a hint?

Think about the shape of the covariance matrix.

Exercise

When should you use BIC over AIC for choosing the number of GMM components?

Need a hint?

Think about the penalty term and dataset size.

Common mistakes

  1. Not enough data: GMMs need sufficient points per component to estimate covariance. Rule of thumb: at least 10d10d points per component (dd = dimensions).
  2. Using full covariance with small data: Full covariance has O(d2)O(d^2) parameters per component. Use diag or spherical for small datasets.
  3. Ignoring initialization: EM converges to local optima. Run multiple times (n_init) and keep the best.

Key takeaways

  • A GMM models data as a weighted mix of Gaussians — soft responsibilities.
  • Unlike K-Means it captures elliptical clusters and probabilistic membership.
  • Covariance type trades flexibility for parameters; full is most expressive.
  • Choose the number of components with BIC/AIC, not the likelihood alone.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelatedGo deeper
Gaussian Mixture ModelsClusteringPCA & Dimensionality Redu…Probability & StatisticsProbability DistributionsBayesian InferenceK-Means ClusteringThe EM AlgorithmHidden Markov ModelsNaive BayesThe ML Project LoopK-Means: Full AlgorithmThe Baum–Welch AlgorithmThe EM Algorithm