What Are Generative Models?Open in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

What Are Generative Models?

Most machine learning is discriminative: given an input xx, predict a label yy. A cat-or-dog classifier learns P(y∣x)P(y \mid x). Generative models instead learn to model the data itself, P(x)P(x), and the fascinating thing through a project-loop lens is that VAEs, GANs, and diffusion are mostly the same hypothesis space (a neural net) with radically different objectives for "make realistic data." Watch the objective slot do the heavy lifting across this course.

Deep dive
The ML Project Loop
Includes runnable notebook

Generative models flip the goal. They learn the distribution of the data itself (P(x)P(x)), so they can sample new examples that look like the originals.

Real-world examples: Image generation (DALL-E, Midjourney), text generation (GPT), drug discovery, data augmentation, anomaly detection, image super-resolution.

Two paradigms

ApproachLearnsExample
DiscriminativeP(y∣x)P(y \mid x): boundary between classesCat vs dog classifier
GenerativeP(x)P(x): what the data looks likeA model that draws new cats and dogs

Intuition

A discriminative model learns to tell real from fake. A generative model learns to be the forger — it creates data so realistic that the discriminative model can't tell the difference.

Fit both to the same points

Same points, two things to learn

probediscriminativeP(orange) = 1.0000totally certaingenerativep(x) = 2.37e-8never seen anything like thistest error, 4000 pointslogistic 2.80%generative 1.93%
training points16
logistic test error2.80%
generative test error1.93%
generative advantage0.88 pts

Both models are fitted live on the sampled points: a logistic regression by gradient descent (dashed boundary) and class-conditional Gaussians in closed form (rings at 1 and 2 standard deviations). Test error is measured on 4000 held-out points. Move the probe to see what each model says about a point it was never shown.

Both models are fitted live on the sampled points, and on this data they draw almost the same boundary. That is the useful surprise: the difference between the paradigms is not the decision, it is everything else the model knows.

Structure substitutes for data. Test error on 4000 held-out points, by training size:

nlogistic regressionclass-conditional Gaussians
84.05%2.08%
162.80%1.93%
302.65%2.00%
601.85%2.02%
4001.82%1.90%

With eight labels the generative model is nearly twice as accurate; by sixty they have converged and the advantage is gone. Modelling P(x,y)P(x,y) commits you to an assumption about the data's shape, and that assumption pays exactly while you have too little data to learn the shape from scratch — the classic Ng & Jordan result.

And only one of them can say "I have never seen this." Move the probe off the edge of the data. At (4.5,3.2)(4.5, 3.2) the logistic regression reports P(orange)=1.0000P(\text{orange}) = 1.0000 (total certainty), while the generative model reports p(x)p(x) on the order of 10−810^{-8}. Push the probe to (−6,5)(-6, 5) and it is P=0.0000P = 0.0000 against p(x)≈10−24p(x) \approx 10^{-24}. A discriminative model has no vocabulary for doubt: it knows which side of a line you are on and literally nothing else, and a point a thousand miles from any training example is simply very far on one side. This is why out-of-distribution detection, anomaly detection and calibration under shift all reach for a density.

Why generate data?

  1. Creative AI: Image synthesis, music composition, text generation
  2. Data augmentation: Generate synthetic training data for rare classes
  3. Anomaly detection: If a new point has low P(x)P(x), it's probably anomalous
  4. Drug discovery: Generate novel molecular structures with desired properties
  5. Compression: Learn efficient representations (VAEs)
  6. Missing data: Fill in missing values by sampling from the learned distribution

How do they learn P(x)P(x)?

1. Explicit density models

Try to directly model P(x)P(x) or a tractable lower bound:

  • VAEs (Variational Autoencoders): learn an approximate posterior over a latent space
  • Normalizing Flows: transform a simple distribution through invertible functions

2. Implicit density models

Learn to sample from P(x)P(x) without ever computing it directly:

  • GANs (Generative Adversarial Networks): a generator fools a discriminator
  • Diffusion Models: iteratively denoise random noise into data

3. Autoregressive models

Factor the joint distribution step-by-step:

P(x)=∏i=1nP(xi∣x1,…,xi−1)P(x) = \prod_{i=1}^{n} P(x_i \mid x_1, \ldots, x_{i-1})
  • RNNs / Transformers for text generation (GPT)
  • PixelCNN for images
Python
# Autoregressive: predict next token given previous ones
# GPT-style: P("cat") = P("c") · P("a"|"c") · P("t"|"ca")

The tradeoff

Explicit models give you exact P(x)P(x) but are often intractable. Implicit models are more flexible but only give you samples, not probabilities. Autoregressive models are exact but slow (sequential generation).

The latent space idea

Most generative models share a common trick: they don't generate data directly. Instead, they learn a latent space — a compressed, lower-dimensional representation where the structure of the data is simpler.

Generate a point in latent space z\mathbf{z}, then decode it into a full data point x\mathbf{x}:

z∼P(z)→decoderx\mathbf{z} \sim P(\mathbf{z}) \quad \xrightarrow{\text{decoder}} \quad \mathbf{x}

The latent space captures the "essence" of the data (style, pose, identity), while the decoder fills in the details.

A concrete density: 2D Gaussian

To make P(x)P(x) concrete, consider a 2D Gaussian with mean μ=[1,2]⊤\mu = [1, 2]^\top and diagonal covariance Σ=diag(1,4)\Sigma = \text{diag}(1, 4). Evaluating the density at the mean x=[1,2]⊤x = [1, 2]^\top (where x−μ=0x - \mu = \mathbf{0}, so the exponent is zero):

p(x)=12π∣Σ∣exp⁡ ⁣(−12(x−μ)⊤Σ−1(x−μ))=12π⋅2exp⁡(0)=14π≈0.0796p(x) = \frac{1}{2\pi\sqrt{|\Sigma|}} \exp\!\left(-\tfrac{1}{2}(x-\mu)^\top \Sigma^{-1}(x-\mu)\right) = \frac{1}{2\pi\cdot 2}\exp(0) = \frac{1}{4\pi} \approx 0.0796

Here ∣Σ∣=1×4=4|\Sigma| = 1 \times 4 = 4, so ∣Σ∣=2\sqrt{|\Sigma|} = 2.

A GAN trained on this distribution can sample x∼p(x)x \sim p(x) but cannot compute p(x)p(x) for a given xx — the generator function G(z)G(z) has no inverse.

Quick comparison

ModelTypeProsCons
VAEExplicitSmooth latent space, stable trainingBlurry outputs
GANImplicitSharp, realistic outputsTraining instability, mode collapse
DiffusionExplicitBest quality, stable trainingSlow sampling
AutoregressiveExplicitExact likelihoodSlow (sequential)

Exercise

Which statement best describes the difference between discriminative and generative models?

Need a hint?

Discriminative models learn boundaries; generative models learn the data distribution.

Exercise

What is the purpose of the latent space in generative models?

Need a hint?

Think about compression and generation.

Exercise

What is the key tradeoff between explicit and implicit generative models?

Need a hint?

Think about what you can compute vs what you can sample.

Common mistakes

  1. Confusing discriminative and generative models: a discriminative model learns p(y∣x)p(y|x); a generative model learns p(x)p(x) or p(x,y)p(x,y). Only generative models can produce new samples.
  2. Misunderstanding latent spaces: the latent vector zz is not the same as learned features in a discriminative model; it is a compressed stochastic representation, not a deterministic encoding.
  3. Equating sampling with evaluating p(x)p(x): GANs can sample but cannot compute p(x)p(x) explicitly; VAEs and diffusion models can (approximately); this distinction matters for density estimation tasks.

Key takeaways

  • Discriminative models learn P(y∣x)P(y\mid x); generative models learn P(x)P(x).
  • Generative models can sample new data, impute, and detect anomalies.
  • Families: explicit density (VAE, autoregressive), implicit (GAN), and diffusion.
  • A latent space is a compact code that samples are decoded from.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelatedGo deeper
What Are Generative Models?Attention & TransformersAutoencodersGaussian Mixture ModelsDiffusion ModelsThe ML Project LoopSampling Methods: Inverse…