Generative Adversarial NetworksOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Generative Adversarial Networks (GANs)

GANs learn to generate realistic data through a competitive game: a generator creates fake samples, and a discriminator tries to tell real from fake. Both improve through competition.

Real-world examples: Image synthesis (StyleGAN), image-to-image translation (pix2pix), super-resolution, data augmentation, art generation.

The game

  • Generator GG: Takes random noise z∼P(z)\mathbf{z} \sim P(\mathbf{z}) and produces fake samples G(z)G(\mathbf{z})
  • Discriminator DD: Classifies inputs as real or fake

The minimax game:

min⁡Gmax⁡D  Ex∼Pdata[log⁡D(x)]+Ez∼P(z)[log⁡(1−D(G(z)))]\min_G \max_D \; \mathbb{E}_{\mathbf{x} \sim P_{\text{data}}}[\log D(\mathbf{x})] + \mathbb{E}_{\mathbf{z} \sim P(\mathbf{z})}[\log(1 - D(G(\mathbf{z})))]

Intuition

The generator is a forger creating fake paintings. The discriminator is an art expert learning to spot fakes. As the expert gets better, the forger must improve. At equilibrium, the forger produces works indistinguishable from real art.

Training dynamics

At Nash equilibrium:

  • GG produces samples indistinguishable from real data
  • D(x)=0.5D(\mathbf{x}) = 0.5 for all x\mathbf{x} — can't tell real from fake

But in practice, training is unstable. Common issues:

ProblemSymptomSolution
Mode collapseGenerator produces limited varietyUnrolled GANs, minibatch discrimination
Training instabilityLoss oscillates wildlySpectral normalization, two-timescale
Vanishing gradientsGenerator doesn't learnNon-saturating loss, Wasserstein
No convergenceNeither network improvesCareful learning rate balancing

GAN Training Trajectory

0.00.20.40.60.8density-4-3-2-10123410.50D(x)Real data p_data(x)Generator p_g(x)
snapshot0 / 20
JSD0.128
progress0%

Blue = real data distribution (bimodal Gaussian). Orange = generator distribution evolving over 20 training snapshots. Thin teal line = discriminator confidence D(x) = P(real). At convergence, D(x) ≈ 0.5 everywhere. Toggle Mode Collapse to see the generator fixate on one mode.

Code: Simple GAN

Python
class Generator(nn.Module):
    def __init__(self, latent_dim=100, img_dim=784):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(latent_dim, 256),
            nn.LeakyReLU(0.2),
            nn.Linear(256, 512),
            nn.LeakyReLU(0.2),
            nn.Linear(512, img_dim),
            nn.Tanh()  # output in [-1, 1]
        )
    
    def forward(self, z):
        return self.net(z)

class Discriminator(nn.Module):
    def __init__(self, img_dim=784):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(img_dim, 512),
            nn.LeakyReLU(0.2),
            nn.Dropout(0.3),
            nn.Linear(512, 256),
            nn.LeakyReLU(0.2),
            nn.Dropout(0.3),
            nn.Linear(256, 1),
            nn.Sigmoid()
        )
    
    def forward(self, x):
        return self.net(x)

# Training
G = Generator()
D = Discriminator()
opt_G = torch.optim.Adam(G.parameters(), lr=2e-4, betas=(0.5, 0.999))
opt_D = torch.optim.Adam(D.parameters(), lr=2e-4, betas=(0.5, 0.999))
criterion = nn.BCELoss()

for epoch in range(100):
    for real_imgs in dataloader:
        batch_size = real_imgs.size(0)
        
        # Train Discriminator
        z = torch.randn(batch_size, 100)
        fake_imgs = G(z).detach()
        
        real_labels = torch.ones(batch_size, 1)
        fake_labels = torch.zeros(batch_size, 1)
        
        d_loss = criterion(D(real_imgs), real_labels) + \
                 criterion(D(fake_imgs), fake_labels)
        
        opt_D.zero_grad()
        d_loss.backward()
        opt_D.step()
        
        # Train Generator
        z = torch.randn(batch_size, 100)
        fake_imgs = G(z)
        g_loss = criterion(D(fake_imgs), real_labels)  # fool discriminator
        
        opt_G.zero_grad()
        g_loss.backward()
        opt_G.step()

Mode collapse

Mode collapse occurs when the generator learns to produce only a few types of outputs that consistently fool the discriminator:

  • Partial mode collapse: Generator produces a subset of modes (e.g., only dogs, no cats)
  • Complete mode collapse: Generator produces the same output for every input

Mode collapse is the #1 GAN problem

If the discriminator is too strong, the generator finds one image that fools it and produces only that. The discriminator can't learn to distinguish because it only sees one type of fake. Solutions: minibatch discrimination, unrolled GANs, Wasserstein loss.

DCGAN (Deep Convolutional GAN)

The first stable GAN architecture for images:

  • Replace pooling with strided convolutions (discriminator) and transposed convolutions (generator)
  • Use batch normalization in both networks
  • Use ReLU in generator (Tanh at output), LeakyReLU in discriminator
Python
class DCGANGenerator(nn.Module):
    def __init__(self, latent_dim=100, channels=3):
        super().__init__()
        self.net = nn.Sequential(
            nn.ConvTranspose2d(latent_dim, 512, 4, 1, 0),
            nn.BatchNorm2d(512), nn.ReLU(),
            nn.ConvTranspose2d(512, 256, 4, 2, 1),
            nn.BatchNorm2d(256), nn.ReLU(),
            nn.ConvTranspose2d(256, 128, 4, 2, 1),
            nn.BatchNorm2d(128), nn.ReLU(),
            nn.ConvTranspose2d(128, channels, 4, 2, 1),
            nn.Tanh()
        )
    
    def forward(self, z):
        return self.net(z.view(-1, 100, 1, 1))

Wasserstein GAN (WGAN)

Replaces the binary cross-entropy with Earth Mover's distance:

min⁡Gmax⁡D∈D  Ex[D(x)]−Ez[D(G(z))]\min_G \max_{D \in \mathcal{D}} \; \mathbb{E}_{\mathbf{x}}[D(\mathbf{x})] - \mathbb{E}_{\mathbf{z}}[D(G(\mathbf{z}))]

Key benefits:

  • Meaningful loss metric (correlates with image quality)
  • More stable training
  • No mode collapse
Python
# WGAN-GP (with gradient penalty)
def gradient_penalty(D, real, fake, device):
    batch_size = real.size(0)
    alpha = torch.rand(batch_size, 1, 1, 1, device=device)
    interpolated = (alpha * real + (1 - alpha) * fake).requires_grad_(True)
    d_interpolated = D(interpolated)
    
    gradients = torch.autograd.grad(
        outputs=d_interpolated, inputs=interpolated,
        grad_outputs=torch.ones_like(d_interpolated),
        create_graph=True, retain_graph=True
    )[0]
    gradients = gradients.view(batch_size, -1)
    penalty = ((gradients.norm(2, dim=1) - 1) ** 2).mean()
    return penalty

GANs vs VAEs

PropertyGANVAE
Output qualitySharp, realisticOften blurry
TrainingUnstableStable
Latent spaceNot structuredSmooth, continuous
GenerationFastFast
Mode coverageMay miss modesCovers all modes

Exercise

In a GAN, what happens at the Nash equilibrium?

Need a hint?

Both players are playing their best response to the other's strategy.

Exercise

What is 'mode collapse' in GANs?

Need a hint?

Think about what happens when the generator finds a shortcut to fool the discriminator.

Exercise

What is the main advantage of Wasserstein GAN (WGAN) over standard GANs?

Need a hint?

Think about training stability and loss interpretation.

Common mistakes

  1. Treating mode collapse as unavoidable: mode collapse is a training instability, not a fundamental limitation; Wasserstein GAN (WGAN) and spectral normalisation substantially reduce it.
  2. Training G and D at the same rate: if D improves much faster than G, the gradient signal vanishes; if G improves faster, D can no longer distinguish real from fake. Balance requires careful learning-rate and update-ratio tuning.
  3. Evaluating GANs by training loss alone: GAN losses do not correlate with sample quality; use FID (Fréchet Inception Distance) or IS (Inception Score) instead.

Key takeaways

  • A GAN pits a generator against a discriminator in a minimax game.
  • At equilibrium the generator matches the data distribution.
  • Mode collapse and unstable training are the main failure modes.
  • WGAN and DCGAN stabilize training; GANs give sharp samples but no likelihood.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelated
Generative Adversarial NetworksAttention & TransformersCNN Visualization & Adver…Variational AutoencodersDiffusion ModelsViT & Modern Generative AI