Diffusion Models
Diffusion models generate data by learning to reverse a gradual noising process. Start with pure noise and iteratively denoise until you get a clean sample. This is the architecture behind DALL-E 2, Stable Diffusion, and Midjourney.
Real-world examples: Image generation (Stable Diffusion), video generation, audio synthesis, 3D shape generation, molecular design.
The two processes
Forward process (fixed)
Gradually add Gaussian noise to the data over timesteps:
After many steps, is pure Gaussian noise. This process is fixed (not learned).
The visualization below applies the forward process to a two-moons dataset — drag the slider or press Play to watch structure dissolve into noise, then switch to Reverse to see it reconstructed:
Diffusion Process — Forward & Reverse
Forward: watch structure dissolve into Gaussian noise over 20 steps. Reverse: watch a denoising network reconstruct structure from noise. Toggle the β schedule to compare linear vs cosine noise addition.
Reverse process (learned)
Train a neural network to reverse each step:
The network learns to predict the noise that was added:
Intuition
Imagine a sculpture buried in sand. The forward process is burying it (adding noise). The reverse process is carefully brushing sand away (denoising) to reveal the sculpture. The neural network learns how to brush — what to remove at each stage.
The training objective
The simplified loss is just the MSE between predicted and actual noise:
where , and defining gives .
Worked number: at a timestep where (roughly the midpoint of the schedule), — the sample is an equal-strength blend of image and noise. Early in the schedule () it's : almost clean. Late (): , nearly pure noise. One formula jumps to any noise level directly — no need to simulate sequential noising steps during training.
import torch
import torch.nn as nn
class SimpleDiffusion(nn.Module):
def __init__(self, img_dim=784, hidden_dim=256, T=1000):
super().__init__()
self.T = T
self.time_embed = nn.Sequential(
nn.Linear(1, hidden_dim), nn.ReLU(), nn.Linear(hidden_dim, hidden_dim)
)
self.net = nn.Sequential(
nn.Linear(img_dim + hidden_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, img_dim)
)
def forward(self, x_t, t):
t_emb = self.time_embed(t.float().unsqueeze(1))
x = torch.cat([x_t, t_emb], dim=1)
return self.net(x)
# Training loop
model = SimpleDiffusion()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
for epoch in range(100):
for x_0 in dataloader:
# Sample random timestep
t = torch.randint(0, model.T, (x_0.size(0),))
# Add noise
noise = torch.randn_like(x_0)
alpha_bar = get_alpha_bar(t) # precomputed schedule
x_t = torch.sqrt(alpha_bar) * x_0 + torch.sqrt(1 - alpha_bar) * noise
# Predict noise
noise_pred = model(x_t, t)
loss = nn.MSELoss()(noise_pred, noise)
optimizer.zero_grad()
loss.backward()
optimizer.step()
Sampling (generation)
Start with and iteratively denoise:
@torch.no_grad()
def sample(model, shape, T=1000):
x = torch.randn(shape) # start from noise
for t in reversed(range(T)):
t_batch = torch.full((shape[0],), t, dtype=torch.long)
noise_pred = model(x, t_batch)
# DDPM update rule
alpha_t = get_alpha(t)
alpha_bar_t = get_alpha_bar(t)
beta_t = get_beta(t)
x = (1 / torch.sqrt(alpha_t)) * (
x - (beta_t / torch.sqrt(1 - alpha_bar_t)) * noise_pred
)
if t > 0:
x += torch.sqrt(beta_t) * torch.randn_like(x)
return x
# Generate 16 images
samples = sample(model, (16, 784))
Noise schedules
The noise schedule controls how fast noise is added:
| Schedule | Formula | Effect |
|---|---|---|
| Linear | Original DDPM | |
| Cosine | , | Smoother, better quality |
Why cosine schedule?
The linear schedule adds noise too quickly in early steps. The cosine schedule is gentler — it keeps the image recognizable longer, giving the model more time to learn meaningful denoising patterns.
Accelerated sampling: DDIM
Standard DDPM requires steps — one network call per noise level. DDIM (Denoising Diffusion Implicit Models) reformulates the reverse process as a deterministic ODE, which makes step-skipping valid: visit only ~50 of the 1000 timesteps (estimate the clean image, re-noise it to the next visited level, repeat) and still land on a high-quality sample. Because the trajectory is deterministic with (the default DDIM setting), the same starting noise always produces the same image — enabling smooth latent-space interpolation. Same trained model, ~20× faster sampling.
Diffusion vs GAN vs VAE
| Property | Diffusion | GAN | VAE |
|---|---|---|---|
| Quality | Best | Very good | Good |
| Diversity | Excellent | Risk of mode collapse | Good |
| Training | Stable | Unstable | Stable |
| Speed | Slow (100-1000 steps) | Fast (1 step) | Fast (1 step) |
| Likelihood | Lower bound | None | Lower bound |
Why diffusion became the dominant paradigm
Diffusion models became the dominant paradigm after 2022 (DALL-E 2, Stable Diffusion, Midjourney) because they combine the training stability of VAEs with the sample quality of GANs — without adversarial training dynamics or posterior collapse. The main trade-off is speed: 50–1000 denoising steps versus a single forward pass. Modern samplers (DDIM, DPM-Solver) and distillation techniques are rapidly closing that gap.
Diffusion models are slow
Generating one image requires 50-1000 forward passes through the network. This is 100-1000× slower than GANs. Acceleration methods (DDIM, distillation, consistency models) are an active research area.
Exercise
In a diffusion model, what does the neural network learn to predict?
Need a hint?
The forward process adds something to the image. The model tries to undo that.
Exercise
Why is the cosine noise schedule preferred over the linear schedule for diffusion models?
Need a hint?
Think about how quickly noise is added in early vs late steps.
Exercise
How does DDIM accelerate diffusion model sampling?
Need a hint?
Think about skipping timesteps.
Common mistakes
- Not enough timesteps: Too few timesteps () gives poor quality. Use at least 1000 for training, 50-100 for DDIM sampling.
- Ignoring the noise schedule: A bad schedule ruins quality. Use cosine schedule for images.
- Forgetting classifier-free guidance: For conditional generation, guidance scale improves quality but reduces diversity. (CFG is covered in the next lesson.)
Key takeaways
- Diffusion adds noise (fixed forward) then learns to denoise (reverse).
- Training predicts the noise added at a random timestep.
- A noise schedule controls the forward corruption.
- DDIM accelerates sampling from ~1000 steps to ~50.
Related concepts
- Generative Adversarial Networks: the GAN approach it overtook
- Variational Autoencoders: another likelihood-style generator
- Neural Radiance Fields (NeRF): a neighboring continuous-representation approach to image synthesis
- Random Walk: the Brownian-motion process whose reverse the forward diffusion corrupts toward