What Are Generative Models?
Most machine learning is discriminative: given an input , predict a label . A cat-or-dog classifier learns . Generative models instead learn to model the data itself, , and the fascinating thing through a project-loop lens is that VAEs, GANs, and diffusion are mostly the same hypothesis space (a neural net) with radically different objectives for "make realistic data." Watch the objective slot do the heavy lifting across this course.
Generative models flip the goal. They learn the distribution of the data itself (), so they can sample new examples that look like the originals.
Real-world examples: Image generation (DALL-E, Midjourney), text generation (GPT), drug discovery, data augmentation, anomaly detection, image super-resolution.
Two paradigms
| Approach | Learns | Example |
|---|---|---|
| Discriminative | : boundary between classes | Cat vs dog classifier |
| Generative | : what the data looks like | A model that draws new cats and dogs |
Intuition
A discriminative model learns to tell real from fake. A generative model learns to be the forger — it creates data so realistic that the discriminative model can't tell the difference.
Fit both to the same points
Same points, two things to learn
Both models are fitted live on the sampled points: a logistic regression by gradient descent (dashed boundary) and class-conditional Gaussians in closed form (rings at 1 and 2 standard deviations). Test error is measured on 4000 held-out points. Move the probe to see what each model says about a point it was never shown.
Both models are fitted live on the sampled points, and on this data they draw almost the same boundary. That is the useful surprise: the difference between the paradigms is not the decision, it is everything else the model knows.
Structure substitutes for data. Test error on 4000 held-out points, by training size:
| n | logistic regression | class-conditional Gaussians |
|---|---|---|
| 8 | 4.05% | 2.08% |
| 16 | 2.80% | 1.93% |
| 30 | 2.65% | 2.00% |
| 60 | 1.85% | 2.02% |
| 400 | 1.82% | 1.90% |
With eight labels the generative model is nearly twice as accurate; by sixty they have converged and the advantage is gone. Modelling commits you to an assumption about the data's shape, and that assumption pays exactly while you have too little data to learn the shape from scratch — the classic Ng & Jordan result.
And only one of them can say "I have never seen this." Move the probe off the edge of the data. At the logistic regression reports (total certainty), while the generative model reports on the order of . Push the probe to and it is against . A discriminative model has no vocabulary for doubt: it knows which side of a line you are on and literally nothing else, and a point a thousand miles from any training example is simply very far on one side. This is why out-of-distribution detection, anomaly detection and calibration under shift all reach for a density.
Why generate data?
- Creative AI: Image synthesis, music composition, text generation
- Data augmentation: Generate synthetic training data for rare classes
- Anomaly detection: If a new point has low , it's probably anomalous
- Drug discovery: Generate novel molecular structures with desired properties
- Compression: Learn efficient representations (VAEs)
- Missing data: Fill in missing values by sampling from the learned distribution
How do they learn ?
1. Explicit density models
Try to directly model or a tractable lower bound:
- VAEs (Variational Autoencoders): learn an approximate posterior over a latent space
- Normalizing Flows: transform a simple distribution through invertible functions
2. Implicit density models
Learn to sample from without ever computing it directly:
- GANs (Generative Adversarial Networks): a generator fools a discriminator
- Diffusion Models: iteratively denoise random noise into data
3. Autoregressive models
Factor the joint distribution step-by-step:
- RNNs / Transformers for text generation (GPT)
- PixelCNN for images
# Autoregressive: predict next token given previous ones
# GPT-style: P("cat") = P("c") · P("a"|"c") · P("t"|"ca")
The tradeoff
Explicit models give you exact but are often intractable. Implicit models are more flexible but only give you samples, not probabilities. Autoregressive models are exact but slow (sequential generation).
The latent space idea
Most generative models share a common trick: they don't generate data directly. Instead, they learn a latent space — a compressed, lower-dimensional representation where the structure of the data is simpler.
Generate a point in latent space , then decode it into a full data point :
The latent space captures the "essence" of the data (style, pose, identity), while the decoder fills in the details.
A concrete density: 2D Gaussian
To make concrete, consider a 2D Gaussian with mean and diagonal covariance . Evaluating the density at the mean (where , so the exponent is zero):
Here , so .
A GAN trained on this distribution can sample but cannot compute for a given — the generator function has no inverse.
Quick comparison
| Model | Type | Pros | Cons |
|---|---|---|---|
| VAE | Explicit | Smooth latent space, stable training | Blurry outputs |
| GAN | Implicit | Sharp, realistic outputs | Training instability, mode collapse |
| Diffusion | Explicit | Best quality, stable training | Slow sampling |
| Autoregressive | Explicit | Exact likelihood | Slow (sequential) |
Exercise
Which statement best describes the difference between discriminative and generative models?
Need a hint?
Discriminative models learn boundaries; generative models learn the data distribution.
Exercise
What is the purpose of the latent space in generative models?
Need a hint?
Think about compression and generation.
Exercise
What is the key tradeoff between explicit and implicit generative models?
Need a hint?
Think about what you can compute vs what you can sample.
Common mistakes
- Confusing discriminative and generative models: a discriminative model learns ; a generative model learns or . Only generative models can produce new samples.
- Misunderstanding latent spaces: the latent vector is not the same as learned features in a discriminative model; it is a compressed stochastic representation, not a deterministic encoding.
- Equating sampling with evaluating : GANs can sample but cannot compute explicitly; VAEs and diffusion models can (approximately); this distinction matters for density estimation tasks.
Key takeaways
- Discriminative models learn ; generative models learn .
- Generative models can sample new data, impute, and detect anomalies.
- Families: explicit density (VAE, autoregressive), implicit (GAN), and diffusion.
- A latent space is a compact code that samples are decoded from.
Related concepts
- Autoencoders: the first step toward learned latent spaces
- Gaussian Mixture Models: a classic explicit-density generative model
- Diffusion Models: the state-of-the-art generative family