Gaussian Mixture Models (GMMs)
K-Means assigns each point to exactly one cluster. GMMs assign soft probabilities — each point belongs to every cluster with some probability. This captures uncertainty and handles elliptical clusters. The shift from K-Means to a mixture of Gaussians is a hypothesis-space change: a richer, probabilistic model family for the same data, and the next lesson's EM is the optimization that fits it.
Real-world examples: Speaker identification, image segmentation, density estimation, anomaly detection, semi-supervised learning.
The model
A GMM assumes data is generated from a mixture of Gaussian distributions:
- : mixing weight (probability of component ),
- : mean of component
- : covariance of component
Intuition
Imagine data comes from three different cameras with different lighting and angles. Each camera produces a Gaussian cloud of points. The GMM figures out which camera took each photo and the settings (mean, spread) of each camera.
Posterior probability (responsibility)
Given a data point , the probability it came from component is:
This is Bayes' theorem — the responsibility of component for point .
EM for Gaussian Mixture Models
Each EM step updates the component means and covariances (M-step) then re-computes soft responsibilities (E-step). Point colors blend between the three component colors weighted by responsibility. Ellipses show the 2σ contour of each Gaussian. Log-likelihood increases monotonically.
Worked example (by hand)
Two 1D components, equal weights , means , both . What is the responsibility of each for the point ?
Plug into the 1D Gaussian (the cancels in the ratio, but we keep it for clarity):
Weight by (both ) and normalize:
So is 92.4% component 1, 7.6% component 2 — a soft assignment. K-Means would hard-assign it 100% to component 1 and throw away that 7.6% of doubt. Points exactly between the means () split .
import numpy as np
from scipy.stats import multivariate_normal
def compute_responsibilities(X, means, covs, weights):
K = len(weights)
responsibilities = np.zeros((len(X), K))
for k in range(K):
responsibilities[:, k] = weights[k] * multivariate_normal.pdf(X, means[k], covs[k])
responsibilities /= responsibilities.sum(axis=1, keepdims=True)
return responsibilities
# Example: 2 components
X = np.array([[1,2], [1.5,2.5], [8,9], [8.5,9.5]])
means = [np.array([1.25, 2.25]), np.array([8.25, 9.25])]
covs = [np.eye(2)*0.1, np.eye(2)*0.1]
weights = [0.5, 0.5]
gamma = compute_responsibilities(X, means, covs, weights)
print("Responsibilities:")
print(gamma.round(3))
# First two points → component 0, last two → component 1
GMM vs K-Means
| Property | K-Means | GMM |
|---|---|---|
| Cluster shape | Spherical | Elliptical |
| Assignment | Hard (0 or 1) | Soft (probability) |
| Parameters | ||
| Optimization | Alternating assign/update | EM algorithm |
| Model selection | Elbow/silhouette | BIC/AIC |
K-Means is a special case of GMM where all covariances are (spherical) and responsibilities are hard-assigned.
Maximum Likelihood Estimation
Parameters are found by maximizing the log-likelihood:
Because of the log-sum (a log of a sum can't be split), there's no closed-form solution. The EM algorithm (next lesson) gets around this by alternating: compute responsibilities (E-step), then update parameters as responsibility-weighted statistics (M-step):
These are exactly the K-Means update equations with the hard 0/1 membership replaced by the soft , which is why a GMM is the soft generalization of K-Means.
Covariance types
scikit-learn supports four covariance structures:
| Type | Parameters per cluster | Shape |
|---|---|---|
full | Arbitrary ellipsoids | |
tied | (mean only) + one shared covariance | Same ellipse for all |
diag | Axis-aligned ellipses | |
spherical | Circles |
(tied shares a single covariance matrix across all clusters, so each cluster only adds its own mean — strictly fewer parameters than full, which is the point of using it.)
from sklearn.mixture import GaussianMixture
# Fit GMM with full covariance
gmm = GaussianMixture(n_components=3, covariance_type='full', random_state=42)
gmm.fit(X)
print(f"Means:\n{gmm.means_}")
print(f"Weights: {gmm.weights_}")
print(f"Log-likelihood: {gmm.score(X) * len(X):.2f}")
# Predict responsibilities
probs = gmm.predict_proba(X)
labels = gmm.predict(X)
Choosing the number of components
BIC (Bayesian Information Criterion)
where is the total number of free parameters in the model (not the number of components ) and is the number of data points. Penalizes complexity more than AIC — prefers simpler models.
AIC (Akaike Information Criterion)
Less penalty than BIC — may select more complex models.
import matplotlib.pyplot as plt
# Enough data to actually compare 1-7 components
rng = np.random.default_rng(42)
X = np.vstack([rng.normal([0, 0], 1.0, (150, 2)),
rng.normal([5, 5], 1.5, (150, 2)),
rng.normal([0, 6], 0.8, (150, 2))]) # 3 true clusters
n_components_range = range(1, 8)
bics = []
aics = []
for n in n_components_range:
gmm = GaussianMixture(n_components=n, random_state=42)
gmm.fit(X)
bics.append(gmm.bic(X))
aics.append(gmm.aic(X))
# Both criteria bottom out at n=3 — the true number of clusters
plt.figure(figsize=(8, 5))
plt.plot(n_components_range, bics, 'bo-', label='BIC')
plt.plot(n_components_range, aics, 'rs-', label='AIC')
plt.xlabel('Number of components')
plt.ylabel('Information criterion')
plt.legend()
plt.title('Choosing K with BIC/AIC')
plt.show()
BIC vs AIC
BIC penalizes complexity more heavily (grows with ). Use BIC when you want a parsimonious model. Use AIC when you suspect the true model is complex. For small datasets, BIC is safer.
Exercise
In a GMM, what does the responsibility γ_ik represent?
Need a hint?
It's a probability given the data and current parameters.
Exercise
In scikit-learn's GMM, what does covariance_type='diag' assume?
Need a hint?
Think about the shape of the covariance matrix.
Exercise
When should you use BIC over AIC for choosing the number of GMM components?
Need a hint?
Think about the penalty term and dataset size.
Common mistakes
- Not enough data: GMMs need sufficient points per component to estimate covariance. Rule of thumb: at least points per component ( = dimensions).
- Using
fullcovariance with small data: Full covariance has parameters per component. Usediagorsphericalfor small datasets. - Ignoring initialization: EM converges to local optima. Run multiple times (
n_init) and keep the best.
Key takeaways
- A GMM models data as a weighted mix of Gaussians — soft responsibilities.
- Unlike K-Means it captures elliptical clusters and probabilistic membership.
- Covariance type trades flexibility for parameters;
fullis most expressive. - Choose the number of components with BIC/AIC, not the likelihood alone.
Related concepts
- The EM Algorithm: the algorithm that fits a GMM
- K-Means Clustering: the hard-assignment special case of a GMM
- Hidden Markov Models: EM applied to sequences