t-SNE & UMAP
PCA finds linear projections. But what if the data lies on a curved manifold? t-SNE and UMAP are non-linear methods that preserve local structure — nearby points stay nearby in the low-dimensional embedding.
Real-world examples: Visualizing single-cell RNA data (thousands of cell types), exploring word embeddings, visualizing deep network activations, cluster discovery in high-dimensional data.
t-SNE (t-distributed Stochastic Neighbor Embedding)
The idea
- Compute pairwise similarities in high-dimensional space using Gaussian kernels
- Initialize random points in 2D
- Compute pairwise similarities in 2D using Student-t distribution (heavier tails)
- Minimize the KL divergence between the two similarity distributions
where = high-D similarities, = low-D similarities.
The actual similarities
High-D (Gaussian). Point "picks neighbor" with conditional probability shrinking with squared distance, bandwidth set per-point (see perplexity below):
The symmetrization makes and guarantees every point contributes (so outliers aren't ignored).
Low-D (Student-t, 1 d.o.f.). The heavy tail is the whole trick — it lets dissimilar points sit far apart cheaply:
Minimizing the KL by gradient descent gives a clean force law — attract when , repel when :
Why Student-t distribution in 2D?
In high dimensions, many points are far apart. In 2D, they'd be forced close together. The Student-t distribution has heavier tails, so distant points in high-D can remain distant in low-D without causing large gradients. This is the key innovation over earlier methods.
Perplexity = effective number of neighbors
Perplexity isn't an arbitrary knob — it's defined as raised to the Shannon entropy of the conditional distribution :
For each point, t-SNE binary-searches until the perplexity hits your target. Interpretation: if spreads evenly over neighbors, so perplexity — the effective neighbor count. Perplexity 30 means "tune each point's Gaussian so it has roughly 30 meaningful neighbors."
Worked mini-example. Suppose puts probability evenly on exactly 4 neighbors ( each, 0 elsewhere). Then , so perplexity — exactly the neighbor count. A peaked distribution like has , perplexity : effectively one neighbor.
- Low perplexity (5-15): small , few effective neighbors → tight local clusters
- High perplexity (30-100): large , many effective neighbors → broader global patterns
- Typical default: 30
t-SNE Perplexity: Effect on Embedding
⚠ t-SNE cluster sizes and inter-cluster distances are NOT meaningful — only topology (which points are neighbors) matters. The same data produces drastically different-looking plots at different perplexity values.
from sklearn.manifold import TSNE
from sklearn.datasets import load_digits
import matplotlib.pyplot as plt
digits = load_digits()
X, y = digits.data, digits.target
# t-SNE embedding
tsne = TSNE(n_components=2, perplexity=30, random_state=42, n_iter=1000)
X_tsne = tsne.fit_transform(X)
plt.figure(figsize=(10, 8))
scatter = plt.scatter(X_tsne[:, 0], X_tsne[:, 1], c=y, cmap='tab10', s=10, alpha=0.7)
plt.colorbar(scatter, label='Digit')
plt.title(f't-SNE (perplexity=30, KL={tsne.kl_divergence_:.3f})')
plt.show()
Limitations of t-SNE
- Slow: in naive implementation (Barnes-Hut approximation: )
- Non-deterministic: Different runs give different embeddings
- Preserves local, not global structure: Cluster sizes and distances between clusters are not meaningful
- Cannot embed new points: No
transform()method — must retrain
Don't over-interpret t-SNE plots
t-SNE cluster sizes, distances between clusters, and global geometry are NOT meaningful. Only the relative positions within clusters are reliable. A tight cluster means the points are similar; that's all you can conclude.
UMAP (Uniform Manifold Approximation and Projection)
UMAP is faster and often better at preserving global structure than t-SNE.
Advantages over t-SNE
| Property | t-SNE | UMAP |
|---|---|---|
| Speed | (Barnes–Hut) | : 10-100× faster |
| Global structure | Poor | Better |
| Transform new data | No | Yes |
| Scalability | ~10K points | ~100K+ points |
Key parameters
- n_neighbors: Local vs global structure (similar to perplexity)
- Small (5-15): tight clusters, local structure
- Large (50-200): broad patterns, global structure
- min_dist: How tightly points pack together
- Small (0.0-0.1): tight clusters
- Large (0.5-1.0): spread out points
import umap
reducer = umap.UMAP(n_neighbors=15, min_dist=0.1, random_state=42)
X_umap = reducer.fit_transform(X)
plt.figure(figsize=(10, 8))
scatter = plt.scatter(X_umap[:, 0], X_umap[:, 1], c=y, cmap='tab10', s=10, alpha=0.7)
plt.colorbar(scatter, label='Digit')
plt.title('UMAP (n_neighbors=15, min_dist=0.1)')
plt.show()
UMAP preserves more global structure
In t-SNE, a cluster of "8"s might appear closer to "3"s than to "6"s just because of local density. UMAP is more likely to preserve the actual relationships — 8s closer to 6s and 3s.
When to use what
| Goal | Method |
|---|---|
| Quick visualization of clusters | UMAP (faster) |
| Publication-quality plots | t-SNE (more established) |
| Embedding new data points | UMAP (has transform) |
| Large datasets (> 100K) | UMAP |
| Preserving global structure | UMAP |
| Very high-dimensional data (images, text) | UMAP with PCA preprocessing |
Recommended pipeline
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
# For high-dimensional data, reduce with PCA first
pipeline = Pipeline([
('scaler', StandardScaler()),
('pca', PCA(n_components=50)), # reduce to 50D first
('umap', umap.UMAP(n_neighbors=15)) # then UMAP to 2D
])
X_embedded = pipeline.fit_transform(X_high_dim)
Exercise
In UMAP, what does increasing 'n_neighbors' do?
Need a hint?
Think about whether you're capturing local or global structure.
Exercise
In t-SNE, what does the perplexity parameter control?
Need a hint?
Think about the balance between local and global structure.
Exercise
Set the t-SNE perplexity for exploring cluster structure in a 3000-point dataset.
Need a hint?
Perplexity ≈ effective number of neighbors. Too low → fractured clusters; too high → washed-out global structure.
Exercise
When should you use UMAP instead of PCA for visualization?
Need a hint?
Think about linear vs non-linear relationships.
Common mistakes
- Using t-SNE/UMAP for clustering: These are visualization tools, not clustering algorithms. Use their output as features for K-Means or DBSCAN, but don't cluster directly on the embedding.
- Interpreting distances literally: Only local neighborhoods are meaningful. A cluster being "larger" or "farther" doesn't mean anything quantitative.
- Not preprocessing: Both methods are sensitive to feature scales. Always standardize first. For high-dimensional data, reduce with PCA first.
- Running on the full dataset: UMAP on 1M+ points is fine, but t-SNE may need subsampling.
Key takeaways
- t-SNE and UMAP are non-linear methods for visualizing high-dim data in 2-D.
- They preserve local neighborhoods; cluster sizes/distances aren't literal.
- t-SNE's perplexity and UMAP's n_neighbors balance local vs global structure.
- Denoise with PCA first; never feed these embeddings to a downstream model.
Related concepts
- Principal Component Analysis: the linear method to run first
- Hierarchical & DBSCAN: clustering the structure these reveal