Variational AutoencodersOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Variational Autoencoders (VAEs)

A VAE is an autoencoder where the latent space is regularized to follow a known distribution — typically a standard Gaussian N(0,I)\mathcal{N}(0, I).

This single change transforms the autoencoder from a compression tool into a generative model.

The key idea

Instead of encoding each input to a point z\mathbf{z}, the encoder outputs the parameters of a distribution:

qϕ(z∣x)=N(μϕ(x),  σϕ2(x)⋅I)q_\phi(\mathbf{z} \mid \mathbf{x}) = \mathcal{N}(\boldsymbol{\mu}_\phi(\mathbf{x}),\; \boldsymbol{\sigma}^2_\phi(\mathbf{x}) \cdot I)

The latent vector is then sampled from this distribution:

z=μ+σ⊙ϵ,ϵ∼N(0,I)\mathbf{z} = \boldsymbol{\mu} + \boldsymbol{\sigma} \odot \boldsymbol{\epsilon}, \quad \boldsymbol{\epsilon} \sim \mathcal{N}(0, I)

Why a distribution, not a point?

By outputting a distribution, the encoder says "this input could map to anywhere near here." This forces nearby latent regions to decode to similar outputs, filling in the "holes" that make standard autoencoders non-generative.

The loss function

The VAE loss has two terms:

L=−log⁡pθ(x∣z)⏟reconstruction+DKL ⁣(qϕ(z∣x)  ∥  N(0,I))⏟KL divergence\mathcal{L} = \underbrace{-\log p_\theta(\mathbf{x}|\mathbf{z})}_{\text{reconstruction}} + \underbrace{D_{\text{KL}}\!\left(q_\phi(\mathbf{z}|\mathbf{x}) \;\|\; \mathcal{N}(0, I)\right)}_{\text{KL divergence}}
  1. Reconstruction loss: measures how well the decoder recovers the input. For a Gaussian decoder this is ∥x−x^∥2\|\mathbf{x} - \hat{\mathbf{x}}\|^2 (MSE); for a Bernoulli decoder (binary data such as black-and-white MNIST pixels) it is binary cross-entropy. The decoder architecture and loss must match.
  2. KL divergence: pulls the encoder's distribution toward the standard Gaussian

The KL divergence has a closed-form solution for two Gaussians:

DKL=−12∑j=1J(1+log⁡σj2−μj2−σj2)D_{\text{KL}} = -\frac{1}{2} \sum_{j=1}^{J} \left(1 + \log \sigma_j^2 - \mu_j^2 - \sigma_j^2\right)

KL divergence intuition

KL divergence measures how different two distributions are. If the encoder outputs N(0,1)\mathcal{N}(0, 1), KL = 0 (matches the prior). If it outputs N(5,1)\mathcal{N}(5, 1), KL is large (penalized). This regularization ensures the latent space is smooth and complete.

Drag the sample point — in AE mode you can land in a "hole" that produces incoherent output; VAE's KL term fills those gaps:

Latent Space: AE vs VAE

AE latent spaceVAE latent space
Interpolated decoder output:
modeVAE
z[0.00, 0.00]

In an AE, the latent space has gaps — sampling a point in a hole produces garbage. A VAE's KL term forces the posterior toward N(0,1), filling the gaps and enabling smooth interpolation.

The reparameterization trick

Why direct sampling breaks backprop

Suppose the encoder outputs μ=0.5\mu = 0.5 and σ=0.3\sigma = 0.3. If we sample z∼N(μ,σ2)\mathbf{z} \sim \mathcal{N}(\mu, \sigma^2) directly, the sample z=0.47\mathbf{z} = 0.47 is just a number — PyTorch cannot compute ∂z/∂μ\partial \mathbf{z} / \partial \mu because the sampling step has no gradient. We are stuck: the decoder loss depends on z\mathbf{z}, but z\mathbf{z} has no gradient path back to the encoder parameters.

The fix: shift the randomness out

Instead of sampling z\mathbf{z} directly, draw noise ϵ∼N(0,I)\boldsymbol{\epsilon} \sim \mathcal{N}(0, I) externally and compute:

z=μ+σ⊙ϵ\mathbf{z} = \boldsymbol{\mu} + \boldsymbol{\sigma} \odot \boldsymbol{\epsilon}

Same example with numbers: μ=0.5\mu = 0.5, σ=0.3\sigma = 0.3, ϵ=−0.1\epsilon = -0.1 (sampled externally):

z=0.5+0.3×(−0.1)=0.47z = 0.5 + 0.3 \times (-0.1) = 0.47

The value of zz is identical. But now zz is a deterministic function of μ\mu and σ\sigma:

∂z∂μ=1,∂z∂σ=ϵ=−0.1\frac{\partial z}{\partial \mu} = 1, \qquad \frac{\partial z}{\partial \sigma} = \epsilon = -0.1

Gradients flow through μ\boldsymbol{\mu} and σ\boldsymbol{\sigma} normally. The randomness lives entirely in ϵ\boldsymbol{\epsilon}, which is treated as a fixed external input — not a learnable parameter.

The reparameterization trick

This is one of the most elegant ideas in deep learning: by shifting the randomness to an external noise variable, we make sampling differentiable. It connects variational inference (a 200-year-old technique) with modern gradient-based optimization.

Code: VAE in PyTorch

Python
class VAE(nn.Module):
    def __init__(self, latent_dim=32):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Linear(784, 256), nn.ReLU(), nn.Linear(256, 128), nn.ReLU()
        )
        self.mu_head = nn.Linear(128, latent_dim)
        self.logvar_head = nn.Linear(128, latent_dim)  # log(σ²) for numerical stability

        self.decoder = nn.Sequential(
            nn.Linear(latent_dim, 128), nn.ReLU(),
            nn.Linear(128, 256), nn.ReLU(),
            nn.Linear(256, 784), nn.Sigmoid()
        )

    def encode(self, x):
        h = self.encoder(x)
        return self.mu_head(h), self.logvar_head(h)

    def reparameterize(self, mu, logvar):
        std = torch.exp(0.5 * logvar)
        eps = torch.randn_like(std)
        return mu + eps * std

    def decode(self, z):
        return self.decoder(z)

    def forward(self, x):
        mu, logvar = self.encode(x)
        z = self.reparameterize(mu, logvar)
        return self.decode(z), mu, logvar

def vae_loss(x_hat, x, mu, logvar):
    # MSE = Gaussian decoder; use F.binary_cross_entropy for binary data (e.g. MNIST)
    recon = F.mse_loss(x_hat, x, reduction='sum')
    kl = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())
    return recon + kl

Sampling from a VAE

Once trained, generating new data is trivial:

Python
# Generate new samples
z = torch.randn(1, latent_dim)   # sample from N(0, I)
x_new = vae.decode(z)            # decode to data space

# Interpolation in latent space
z1 = torch.randn(1, latent_dim)
z2 = torch.randn(1, latent_dim)
for alpha in torch.linspace(0, 1, 10):
    z = (1 - alpha) * z1 + alpha * z2
    x_interp = vae.decode(z)  # smooth transition!

VAE vs standard autoencoder

PropertyAutoencoderVAE
Latent spaceDiscontinuous (holes)Continuous (smooth)
Latent representationPointDistribution
GenerationCannot generateCan generate
LossReconstruction onlyReconstruction + KL
SamplingDeterministicStochastic (reparameterized)

The β-VAE

Adding a weight β\beta to the KL term controls the tradeoff between reconstruction quality and latent space regularization:

L=reconstruction+β⋅DKL\mathcal{L} = \text{reconstruction} + \beta \cdot D_{\text{KL}}
  • β<1\beta < 1: better reconstruction, less structured latent space
  • β>1\beta > 1: encourages more disentangled representations, but with too large a β\beta the encoder may ignore the input entirely (posterior collapse), producing blurry or degenerate outputs

Exercise

In a VAE, what does the KL divergence term in the loss function do?

Need a hint?

It compares the encoder's output distribution to a target distribution.

Exercise

Why is the reparameterization trick necessary in VAEs?

Need a hint?

Can you backpropagate through a random sampling operation?

Exercise

In a β-VAE, what happens when you increase β above 1?

Need a hint?

β weights the KL divergence term.

Exercise

Set β in a β-VAE to encourage disentangled representations while keeping reconstructions reasonable.

0.110.0520
Need a hint?

β = 1 is a standard VAE. Higher β forces more independent latent dimensions at the cost of reconstruction quality.

Common mistakes

  1. KL divergence too high or too low: if β\beta (the KL weight) is too large the model ignores the input ("posterior collapse"); too small and it ignores the prior and cannot sample coherently.
  2. Misunderstanding the reparameterisation trick: sampling z∼N(μ,σ2)z \sim \mathcal{N}(\mu, \sigma^2) is not differentiable; the trick rewrites this as z=μ+σϵz = \mu + \sigma \epsilon where ϵ∼N(0,1)\epsilon \sim \mathcal{N}(0,1), moving randomness outside the gradient path.
  3. Treating the ELBO bound as the true log-likelihood: the ELBO is a lower bound; tighter bounds (importance-weighted ELBO) exist but require more samples.

Key takeaways

  • A VAE encodes to a distribution, regularized toward N(0,I)\mathcal{N}(0,I) by KL.
  • Loss = reconstruction + KL divergence.
  • The reparameterization trick makes sampling differentiable.
  • The smooth latent space supports sampling and interpolation; β tunes disentanglement.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelatedGo deeper
Variational AutoencodersAttention & TransformersEntropy & KL DivergenceAutoencodersGenerative Adversarial Ne…The EM AlgorithmDiffusion ModelsViT & Modern Generative AIDDIM Sampling