Diffusion ModelsOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Diffusion Models

Diffusion models generate data by learning to reverse a gradual noising process. Start with pure noise and iteratively denoise until you get a clean sample. This is the architecture behind DALL-E 2, Stable Diffusion, and Midjourney.

Real-world examples: Image generation (Stable Diffusion), video generation, audio synthesis, 3D shape generation, molecular design.

The two processes

Forward process (fixed)

Gradually add Gaussian noise to the data over TT timesteps:

q(xt∣xt−1)=N(xt;1−βt xt−1,βtI)q(\mathbf{x}_t \mid \mathbf{x}_{t-1}) = \mathcal{N}(\mathbf{x}_t; \sqrt{1-\beta_t} \, \mathbf{x}_{t-1}, \beta_t \mathbf{I})

After many steps, xT\mathbf{x}_T is pure Gaussian noise. This process is fixed (not learned).

The visualization below applies the forward process to a two-moons dataset — drag the slider or press Play to watch structure dissolve into noise, then switch to Reverse to see it reconstructed:

Diffusion Process — Forward & Reverse

modeForward
t0
β_t—
σ_t—

Forward: watch structure dissolve into Gaussian noise over 20 steps. Reverse: watch a denoising network reconstruct structure from noise. Toggle the β schedule to compare linear vs cosine noise addition.

Reverse process (learned)

Train a neural network to reverse each step:

pθ(xt−1∣xt)=N(xt−1;μθ(xt,t),σt2I)p_\theta(\mathbf{x}_{t-1} \mid \mathbf{x}_t) = \mathcal{N}(\mathbf{x}_{t-1}; \boldsymbol{\mu}_\theta(\mathbf{x}_t, t), \sigma_t^2 \mathbf{I})

The network learns to predict the noise ϵ\boldsymbol{\epsilon} that was added:

ϵθ(xt,t)≈ϵ\boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t) \approx \boldsymbol{\epsilon}

Intuition

Imagine a sculpture buried in sand. The forward process is burying it (adding noise). The reverse process is carefully brushing sand away (denoising) to reveal the sculpture. The neural network learns how to brush — what to remove at each stage.

The training objective

The simplified loss is just the MSE between predicted and actual noise:

L=Et,x0,ϵ[∥ϵ−ϵθ(xt,t)∥2]\mathcal{L} = \mathbb{E}_{t, \mathbf{x}_0, \boldsymbol{\epsilon}} \left[ \| \boldsymbol{\epsilon} - \boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t) \|^2 \right]

where xt=αˉt x0+1−αˉt ϵ\mathbf{x}_t = \sqrt{\bar{\alpha}_t} \, \mathbf{x}_0 + \sqrt{1 - \bar{\alpha}_t} \, \boldsymbol{\epsilon}, and defining αt=1−βt\alpha_t = 1 - \beta_t gives αˉt=∏s=1tαs\bar{\alpha}_t = \prod_{s=1}^t \alpha_s.

Worked number: at a timestep where αˉt=0.5\bar{\alpha}_t = 0.5 (roughly the midpoint of the schedule), xt=0.5 x0+0.5 ϵ≈0.707 x0+0.707 ϵ\mathbf{x}_t = \sqrt{0.5}\,\mathbf{x}_0 + \sqrt{0.5}\,\boldsymbol{\epsilon} \approx 0.707\,\mathbf{x}_0 + 0.707\,\boldsymbol{\epsilon} — the sample is an equal-strength blend of image and noise. Early in the schedule (αˉt≈0.99\bar{\alpha}_t \approx 0.99) it's 0.995 x0+0.1 ϵ0.995\,\mathbf{x}_0 + 0.1\,\boldsymbol{\epsilon}: almost clean. Late (αˉt≈0.01\bar{\alpha}_t \approx 0.01): 0.1 x0+0.995 ϵ0.1\,\mathbf{x}_0 + 0.995\,\boldsymbol{\epsilon}, nearly pure noise. One formula jumps to any noise level directly — no need to simulate tt sequential noising steps during training.

Python
import torch
import torch.nn as nn

class SimpleDiffusion(nn.Module):
    def __init__(self, img_dim=784, hidden_dim=256, T=1000):
        super().__init__()
        self.T = T
        self.time_embed = nn.Sequential(
            nn.Linear(1, hidden_dim), nn.ReLU(), nn.Linear(hidden_dim, hidden_dim)
        )
        self.net = nn.Sequential(
            nn.Linear(img_dim + hidden_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, img_dim)
        )
    
    def forward(self, x_t, t):
        t_emb = self.time_embed(t.float().unsqueeze(1))
        x = torch.cat([x_t, t_emb], dim=1)
        return self.net(x)

# Training loop
model = SimpleDiffusion()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

for epoch in range(100):
    for x_0 in dataloader:
        # Sample random timestep
        t = torch.randint(0, model.T, (x_0.size(0),))
        
        # Add noise
        noise = torch.randn_like(x_0)
        alpha_bar = get_alpha_bar(t)  # precomputed schedule
        x_t = torch.sqrt(alpha_bar) * x_0 + torch.sqrt(1 - alpha_bar) * noise
        
        # Predict noise
        noise_pred = model(x_t, t)
        loss = nn.MSELoss()(noise_pred, noise)
        
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

Sampling (generation)

Start with xT∼N(0,I)\mathbf{x}_T \sim \mathcal{N}(0, I) and iteratively denoise:

Python
@torch.no_grad()
def sample(model, shape, T=1000):
    x = torch.randn(shape)  # start from noise
    
    for t in reversed(range(T)):
        t_batch = torch.full((shape[0],), t, dtype=torch.long)
        noise_pred = model(x, t_batch)
        
        # DDPM update rule
        alpha_t = get_alpha(t)
        alpha_bar_t = get_alpha_bar(t)
        beta_t = get_beta(t)
        
        x = (1 / torch.sqrt(alpha_t)) * (
            x - (beta_t / torch.sqrt(1 - alpha_bar_t)) * noise_pred
        )
        
        if t > 0:
            x += torch.sqrt(beta_t) * torch.randn_like(x)
    
    return x

# Generate 16 images
samples = sample(model, (16, 784))

Noise schedules

The noise schedule βt\beta_t controls how fast noise is added:

ScheduleFormulaEffect
Linearβt=β1+(t−1)βT−β1T−1\beta_t = \beta_1 + (t-1) \dfrac{\beta_T - \beta_1}{T-1}Original DDPM
Cosineαˉt=f(t)f(0)\bar{\alpha}_t = \frac{f(t)}{f(0)}, f(t)=cos⁡(t/T+s1+s⋅π2)2f(t) = \cos\left(\frac{t/T + s}{1+s} \cdot \frac{\pi}{2}\right)^2Smoother, better quality

Why cosine schedule?

The linear schedule adds noise too quickly in early steps. The cosine schedule is gentler — it keeps the image recognizable longer, giving the model more time to learn meaningful denoising patterns.

Accelerated sampling: DDIM

Standard DDPM requires T=1000T=1000 steps — one network call per noise level. DDIM (Denoising Diffusion Implicit Models) reformulates the reverse process as a deterministic ODE, which makes step-skipping valid: visit only ~50 of the 1000 timesteps (estimate the clean image, re-noise it to the next visited level, repeat) and still land on a high-quality sample. Because the trajectory is deterministic with η=0\eta = 0 (the default DDIM setting), the same starting noise always produces the same image — enabling smooth latent-space interpolation. Same trained model, ~20× faster sampling.

Deep dive
DDIM Sampling
Includes runnable notebook

Diffusion vs GAN vs VAE

PropertyDiffusionGANVAE
QualityBestVery goodGood
DiversityExcellentRisk of mode collapseGood
TrainingStableUnstableStable
SpeedSlow (100-1000 steps)Fast (1 step)Fast (1 step)
LikelihoodLower boundNoneLower bound

Why diffusion became the dominant paradigm

Diffusion models became the dominant paradigm after 2022 (DALL-E 2, Stable Diffusion, Midjourney) because they combine the training stability of VAEs with the sample quality of GANs — without adversarial training dynamics or posterior collapse. The main trade-off is speed: 50–1000 denoising steps versus a single forward pass. Modern samplers (DDIM, DPM-Solver) and distillation techniques are rapidly closing that gap.

Diffusion models are slow

Generating one image requires 50-1000 forward passes through the network. This is 100-1000× slower than GANs. Acceleration methods (DDIM, distillation, consistency models) are an active research area.

Exercise

In a diffusion model, what does the neural network learn to predict?

Need a hint?

The forward process adds something to the image. The model tries to undo that.

Exercise

Why is the cosine noise schedule preferred over the linear schedule for diffusion models?

Need a hint?

Think about how quickly noise is added in early vs late steps.

Exercise

How does DDIM accelerate diffusion model sampling?

Need a hint?

Think about skipping timesteps.

Common mistakes

  1. Not enough timesteps: Too few timesteps (T<100T < 100) gives poor quality. Use at least 1000 for training, 50-100 for DDIM sampling.
  2. Ignoring the noise schedule: A bad schedule ruins quality. Use cosine schedule for images.
  3. Forgetting classifier-free guidance: For conditional generation, guidance scale w>1w > 1 improves quality but reduces diversity. (CFG is covered in the next lesson.)

Key takeaways

  • Diffusion adds noise (fixed forward) then learns to denoise (reverse).
  • Training predicts the noise added at a random timestep.
  • A noise schedule controls the forward corruption.
  • DDIM accelerates sampling from ~1000 steps to ~50.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelatedGo deeper
Diffusion ModelsAttention & TransformersWhat Are Generative Model…Generative Adversarial Ne…ViT & Modern Generative AIVariational AutoencodersVoice & Multimodal AINeural Radiance Fields (N…Random WalkDDIM Sampling