Generative Adversarial Networks (GANs)
GANs learn to generate realistic data through a competitive game: a generator creates fake samples, and a discriminator tries to tell real from fake. Both improve through competition.
Real-world examples: Image synthesis (StyleGAN), image-to-image translation (pix2pix), super-resolution, data augmentation, art generation.
The game
- Generator : Takes random noise and produces fake samples
- Discriminator : Classifies inputs as real or fake
The minimax game:
Intuition
The generator is a forger creating fake paintings. The discriminator is an art expert learning to spot fakes. As the expert gets better, the forger must improve. At equilibrium, the forger produces works indistinguishable from real art.
Training dynamics
At Nash equilibrium:
- produces samples indistinguishable from real data
- for all — can't tell real from fake
But in practice, training is unstable. Common issues:
| Problem | Symptom | Solution |
|---|---|---|
| Mode collapse | Generator produces limited variety | Unrolled GANs, minibatch discrimination |
| Training instability | Loss oscillates wildly | Spectral normalization, two-timescale |
| Vanishing gradients | Generator doesn't learn | Non-saturating loss, Wasserstein |
| No convergence | Neither network improves | Careful learning rate balancing |
GAN Training Trajectory
Blue = real data distribution (bimodal Gaussian). Orange = generator distribution evolving over 20 training snapshots. Thin teal line = discriminator confidence D(x) = P(real). At convergence, D(x) ≈ 0.5 everywhere. Toggle Mode Collapse to see the generator fixate on one mode.
Code: Simple GAN
class Generator(nn.Module):
def __init__(self, latent_dim=100, img_dim=784):
super().__init__()
self.net = nn.Sequential(
nn.Linear(latent_dim, 256),
nn.LeakyReLU(0.2),
nn.Linear(256, 512),
nn.LeakyReLU(0.2),
nn.Linear(512, img_dim),
nn.Tanh() # output in [-1, 1]
)
def forward(self, z):
return self.net(z)
class Discriminator(nn.Module):
def __init__(self, img_dim=784):
super().__init__()
self.net = nn.Sequential(
nn.Linear(img_dim, 512),
nn.LeakyReLU(0.2),
nn.Dropout(0.3),
nn.Linear(512, 256),
nn.LeakyReLU(0.2),
nn.Dropout(0.3),
nn.Linear(256, 1),
nn.Sigmoid()
)
def forward(self, x):
return self.net(x)
# Training
G = Generator()
D = Discriminator()
opt_G = torch.optim.Adam(G.parameters(), lr=2e-4, betas=(0.5, 0.999))
opt_D = torch.optim.Adam(D.parameters(), lr=2e-4, betas=(0.5, 0.999))
criterion = nn.BCELoss()
for epoch in range(100):
for real_imgs in dataloader:
batch_size = real_imgs.size(0)
# Train Discriminator
z = torch.randn(batch_size, 100)
fake_imgs = G(z).detach()
real_labels = torch.ones(batch_size, 1)
fake_labels = torch.zeros(batch_size, 1)
d_loss = criterion(D(real_imgs), real_labels) + \
criterion(D(fake_imgs), fake_labels)
opt_D.zero_grad()
d_loss.backward()
opt_D.step()
# Train Generator
z = torch.randn(batch_size, 100)
fake_imgs = G(z)
g_loss = criterion(D(fake_imgs), real_labels) # fool discriminator
opt_G.zero_grad()
g_loss.backward()
opt_G.step()
Mode collapse
Mode collapse occurs when the generator learns to produce only a few types of outputs that consistently fool the discriminator:
- Partial mode collapse: Generator produces a subset of modes (e.g., only dogs, no cats)
- Complete mode collapse: Generator produces the same output for every input
Mode collapse is the #1 GAN problem
If the discriminator is too strong, the generator finds one image that fools it and produces only that. The discriminator can't learn to distinguish because it only sees one type of fake. Solutions: minibatch discrimination, unrolled GANs, Wasserstein loss.
DCGAN (Deep Convolutional GAN)
The first stable GAN architecture for images:
- Replace pooling with strided convolutions (discriminator) and transposed convolutions (generator)
- Use batch normalization in both networks
- Use ReLU in generator (Tanh at output), LeakyReLU in discriminator
class DCGANGenerator(nn.Module):
def __init__(self, latent_dim=100, channels=3):
super().__init__()
self.net = nn.Sequential(
nn.ConvTranspose2d(latent_dim, 512, 4, 1, 0),
nn.BatchNorm2d(512), nn.ReLU(),
nn.ConvTranspose2d(512, 256, 4, 2, 1),
nn.BatchNorm2d(256), nn.ReLU(),
nn.ConvTranspose2d(256, 128, 4, 2, 1),
nn.BatchNorm2d(128), nn.ReLU(),
nn.ConvTranspose2d(128, channels, 4, 2, 1),
nn.Tanh()
)
def forward(self, z):
return self.net(z.view(-1, 100, 1, 1))
Wasserstein GAN (WGAN)
Replaces the binary cross-entropy with Earth Mover's distance:
Key benefits:
- Meaningful loss metric (correlates with image quality)
- More stable training
- No mode collapse
# WGAN-GP (with gradient penalty)
def gradient_penalty(D, real, fake, device):
batch_size = real.size(0)
alpha = torch.rand(batch_size, 1, 1, 1, device=device)
interpolated = (alpha * real + (1 - alpha) * fake).requires_grad_(True)
d_interpolated = D(interpolated)
gradients = torch.autograd.grad(
outputs=d_interpolated, inputs=interpolated,
grad_outputs=torch.ones_like(d_interpolated),
create_graph=True, retain_graph=True
)[0]
gradients = gradients.view(batch_size, -1)
penalty = ((gradients.norm(2, dim=1) - 1) ** 2).mean()
return penalty
GANs vs VAEs
| Property | GAN | VAE |
|---|---|---|
| Output quality | Sharp, realistic | Often blurry |
| Training | Unstable | Stable |
| Latent space | Not structured | Smooth, continuous |
| Generation | Fast | Fast |
| Mode coverage | May miss modes | Covers all modes |
Exercise
In a GAN, what happens at the Nash equilibrium?
Need a hint?
Both players are playing their best response to the other's strategy.
Exercise
What is 'mode collapse' in GANs?
Need a hint?
Think about what happens when the generator finds a shortcut to fool the discriminator.
Exercise
What is the main advantage of Wasserstein GAN (WGAN) over standard GANs?
Need a hint?
Think about training stability and loss interpretation.
Common mistakes
- Treating mode collapse as unavoidable: mode collapse is a training instability, not a fundamental limitation; Wasserstein GAN (WGAN) and spectral normalisation substantially reduce it.
- Training G and D at the same rate: if D improves much faster than G, the gradient signal vanishes; if G improves faster, D can no longer distinguish real from fake. Balance requires careful learning-rate and update-ratio tuning.
- Evaluating GANs by training loss alone: GAN losses do not correlate with sample quality; use FID (Fréchet Inception Distance) or IS (Inception Score) instead.
Key takeaways
- A GAN pits a generator against a discriminator in a minimax game.
- At equilibrium the generator matches the data distribution.
- Mode collapse and unstable training are the main failure modes.
- WGAN and DCGAN stabilize training; GANs give sharp samples but no likelihood.
Related concepts
- Variational Autoencoders: the likelihood-based alternative
- Diffusion Models: what largely overtook GANs for image quality