Variational Autoencoders (VAEs)
A VAE is an autoencoder where the latent space is regularized to follow a known distribution — typically a standard Gaussian .
This single change transforms the autoencoder from a compression tool into a generative model.
The key idea
Instead of encoding each input to a point , the encoder outputs the parameters of a distribution:
The latent vector is then sampled from this distribution:
Why a distribution, not a point?
By outputting a distribution, the encoder says "this input could map to anywhere near here." This forces nearby latent regions to decode to similar outputs, filling in the "holes" that make standard autoencoders non-generative.
The loss function
The VAE loss has two terms:
- Reconstruction loss: measures how well the decoder recovers the input. For a Gaussian decoder this is (MSE); for a Bernoulli decoder (binary data such as black-and-white MNIST pixels) it is binary cross-entropy. The decoder architecture and loss must match.
- KL divergence: pulls the encoder's distribution toward the standard Gaussian
The KL divergence has a closed-form solution for two Gaussians:
KL divergence intuition
KL divergence measures how different two distributions are. If the encoder outputs , KL = 0 (matches the prior). If it outputs , KL is large (penalized). This regularization ensures the latent space is smooth and complete.
Drag the sample point — in AE mode you can land in a "hole" that produces incoherent output; VAE's KL term fills those gaps:
Latent Space: AE vs VAE
In an AE, the latent space has gaps — sampling a point in a hole produces garbage. A VAE's KL term forces the posterior toward N(0,1), filling the gaps and enabling smooth interpolation.
The reparameterization trick
Why direct sampling breaks backprop
Suppose the encoder outputs and . If we sample directly, the sample is just a number — PyTorch cannot compute because the sampling step has no gradient. We are stuck: the decoder loss depends on , but has no gradient path back to the encoder parameters.
The fix: shift the randomness out
Instead of sampling directly, draw noise externally and compute:
Same example with numbers: , , (sampled externally):
The value of is identical. But now is a deterministic function of and :
Gradients flow through and normally. The randomness lives entirely in , which is treated as a fixed external input — not a learnable parameter.
The reparameterization trick
This is one of the most elegant ideas in deep learning: by shifting the randomness to an external noise variable, we make sampling differentiable. It connects variational inference (a 200-year-old technique) with modern gradient-based optimization.
Code: VAE in PyTorch
class VAE(nn.Module):
def __init__(self, latent_dim=32):
super().__init__()
self.encoder = nn.Sequential(
nn.Linear(784, 256), nn.ReLU(), nn.Linear(256, 128), nn.ReLU()
)
self.mu_head = nn.Linear(128, latent_dim)
self.logvar_head = nn.Linear(128, latent_dim) # log(σ²) for numerical stability
self.decoder = nn.Sequential(
nn.Linear(latent_dim, 128), nn.ReLU(),
nn.Linear(128, 256), nn.ReLU(),
nn.Linear(256, 784), nn.Sigmoid()
)
def encode(self, x):
h = self.encoder(x)
return self.mu_head(h), self.logvar_head(h)
def reparameterize(self, mu, logvar):
std = torch.exp(0.5 * logvar)
eps = torch.randn_like(std)
return mu + eps * std
def decode(self, z):
return self.decoder(z)
def forward(self, x):
mu, logvar = self.encode(x)
z = self.reparameterize(mu, logvar)
return self.decode(z), mu, logvar
def vae_loss(x_hat, x, mu, logvar):
# MSE = Gaussian decoder; use F.binary_cross_entropy for binary data (e.g. MNIST)
recon = F.mse_loss(x_hat, x, reduction='sum')
kl = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())
return recon + kl
Sampling from a VAE
Once trained, generating new data is trivial:
# Generate new samples
z = torch.randn(1, latent_dim) # sample from N(0, I)
x_new = vae.decode(z) # decode to data space
# Interpolation in latent space
z1 = torch.randn(1, latent_dim)
z2 = torch.randn(1, latent_dim)
for alpha in torch.linspace(0, 1, 10):
z = (1 - alpha) * z1 + alpha * z2
x_interp = vae.decode(z) # smooth transition!
VAE vs standard autoencoder
| Property | Autoencoder | VAE |
|---|---|---|
| Latent space | Discontinuous (holes) | Continuous (smooth) |
| Latent representation | Point | Distribution |
| Generation | Cannot generate | Can generate |
| Loss | Reconstruction only | Reconstruction + KL |
| Sampling | Deterministic | Stochastic (reparameterized) |
The β-VAE
Adding a weight to the KL term controls the tradeoff between reconstruction quality and latent space regularization:
- : better reconstruction, less structured latent space
- : encourages more disentangled representations, but with too large a the encoder may ignore the input entirely (posterior collapse), producing blurry or degenerate outputs
Exercise
In a VAE, what does the KL divergence term in the loss function do?
Need a hint?
It compares the encoder's output distribution to a target distribution.
Exercise
Why is the reparameterization trick necessary in VAEs?
Need a hint?
Can you backpropagate through a random sampling operation?
Exercise
In a β-VAE, what happens when you increase β above 1?
Need a hint?
β weights the KL divergence term.
Exercise
Set β in a β-VAE to encourage disentangled representations while keeping reconstructions reasonable.
Need a hint?
β = 1 is a standard VAE. Higher β forces more independent latent dimensions at the cost of reconstruction quality.
Common mistakes
- KL divergence too high or too low: if (the KL weight) is too large the model ignores the input ("posterior collapse"); too small and it ignores the prior and cannot sample coherently.
- Misunderstanding the reparameterisation trick: sampling is not differentiable; the trick rewrites this as where , moving randomness outside the gradient path.
- Treating the ELBO bound as the true log-likelihood: the ELBO is a lower bound; tighter bounds (importance-weighted ELBO) exist but require more samples.
Key takeaways
- A VAE encodes to a distribution, regularized toward by KL.
- Loss = reconstruction + KL divergence.
- The reparameterization trick makes sampling differentiable.
- The smooth latent space supports sampling and interpolation; β tunes disentanglement.
Related concepts
- Autoencoders: the deterministic autoencoder it upgrades
- Generative Adversarial Networks: the adversarial alternative
- The EM Algorithm: the latent-variable likelihood view