AutoencodersOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Autoencoders

An autoencoder learns to compress data into a compact representation and reconstruct it. The bottleneck forces the network to learn the most important features.

Real-world examples: Image denoising, dimensionality reduction, anomaly detection, feature learning, data compression.

Architecture

An autoencoder has two parts:

  • Encoder: Maps input x\mathbf{x} to a latent code z=fenc(x)\mathbf{z} = f_{\text{enc}}(\mathbf{x})
  • Decoder: Reconstructs from the code x^=fdec(z)\hat{\mathbf{x}} = f_{\text{dec}}(\mathbf{z})

The training objective is to minimize reconstruction error:

L=∥x−x^∥2\mathcal{L} = \|\mathbf{x} - \hat{\mathbf{x}}\|^2

Intuition

Think of autoencoders as a data compression algorithm. The encoder compresses, the decoder decompresses. The bottleneck is like a ZIP file. It can only store the essential information.

Code: Simple autoencoder

Python
import torch
import torch.nn as nn

class Autoencoder(nn.Module):
    def __init__(self, input_dim=784, latent_dim=32):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Linear(input_dim, 256),
            nn.ReLU(),
            nn.Linear(256, 128),
            nn.ReLU(),
            nn.Linear(128, latent_dim)
        )
        self.decoder = nn.Sequential(
            nn.Linear(latent_dim, 128),
            nn.ReLU(),
            nn.Linear(128, 256),
            nn.ReLU(),
            nn.Linear(256, input_dim),
            nn.Sigmoid()
        )
    
    def forward(self, x):
        z = self.encoder(x)
        x_hat = self.decoder(z)
        return x_hat

# Training loop
model = Autoencoder(input_dim=784, latent_dim=32)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

for epoch in range(10):
    x_hat = model(x)
    loss = nn.MSELoss()(x_hat, x)  # mean over pixels — matches nn.MSELoss(reduction='mean')
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
    print(f"Epoch {epoch}: loss={loss.item():.4f}")

What the bottleneck actually buys

Where the error stops falling is the intrinsic dimension

0.9800code size 16reconstruction MSEeigenvalues (kept in teal)
code size3
reconstruction MSE0.05215
variance kept94.68%
improvement from the last unit55.3%
noise floor σ²0.06250

Three latent directions embedded in 16 dimensions plus isotropic noise. A linear autoencoder trained to convergence is PCA, so reconstruction error at code size k is exactly the tail of the eigenvalue spectrum — computed here by diagonalising the sample covariance, not by training. Bars are the eigenvalues; the curve is reconstruction MSE.

A linear autoencoder trained to convergence is PCA, so reconstruction error at code size kk is exactly the tail of the eigenvalue spectrum, which means it can be computed rather than trained, with no question about whether the optimiser converged. The data here is 3 latent directions embedded in 16 dimensions plus isotropic noise:

code sizereconstruction MSEvariance kept
10.3527364.04%
20.1166788.11%
30.0521594.68%
40.0470895.20%
80.0287597.07%

Going from 2 to 3 more than halves the error. Going from 3 to 4 improves it by 10%, and every unit after that buys a slice of noise. The eigenvalue bars say it more bluntly (10.049, 3.777, 1.03210.049,\ 3.777,\ 1.032, then 0.081, 0.077, 0.0760.081,\ 0.077,\ 0.076): a cliff exactly at the intrinsic dimension.

That elbow is the whole argument for the bottleneck. Set the code size at or above the data's real dimensionality and the network can afford to learn the identity map: perfect reconstruction, zero representation. Set it below and the model is forced to choose what to keep, and the point where the error stops falling tells you how many directions the data actually had. A "compression" framing undersells this: you are not trying to make the file small, you are using scarcity to force a decision about structure.

Types of autoencoders

Undercomplete autoencoder

Latent dimension < input dimension. Forces compression. Most basic form.

Sparse autoencoder

Adds a sparsity penalty to the latent space:

L=∥x−x^∥2+λ∑j∣zj∣\mathcal{L} = \|\mathbf{x} - \hat{\mathbf{x}}\|^2 + \lambda \sum_j |z_j|

Denoising autoencoder

Trains on corrupted inputs, reconstructs clean ones:

Python
def add_noise(x, noise_factor=0.3):
    return x + noise_factor * torch.randn_like(x)

# Train on noisy input, reconstruct clean input
x_noisy = add_noise(x)
x_hat = model(x_noisy)
loss = nn.MSELoss()(x_hat, x)  # compare to CLEAN input

Convolutional autoencoder

Uses Conv2d layers for images:

Python
class ConvAutoencoder(nn.Module):
    def __init__(self):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Conv2d(1, 16, 3, stride=2, padding=1),  # 28→14
            nn.ReLU(),
            nn.Conv2d(16, 32, 3, stride=2, padding=1), # 14→7
            nn.ReLU(),
            nn.Conv2d(32, 64, 7),                       # 7→1
        )
        self.decoder = nn.Sequential(
            nn.ConvTranspose2d(64, 32, 7),
            nn.ReLU(),
            nn.ConvTranspose2d(32, 16, 3, stride=2, padding=1, output_padding=1),
            nn.ReLU(),
            nn.ConvTranspose2d(16, 1, 3, stride=2, padding=1, output_padding=1),
            nn.Sigmoid()
        )
    
    def forward(self, x):
        z = self.encoder(x)
        return self.decoder(z)

Why autoencoders aren't generative

A standard autoencoder maps each input to a single point in latent space. The latent space has "holes" — regions never seen during training. If you sample randomly from the latent space, the decoder produces garbage.

The holes problem

Imagine the encoder maps all training images to a circle in 2D latent space. Points inside or outside the circle were never decoded. Sampling randomly gives unrealistic outputs. VAEs solve this by forcing the latent space to be continuous and complete.

Applications

ApplicationHow
Anomalous data detectionHigh reconstruction error → anomaly
Dimensionality reductionUse encoder output as features
Image denoisingTrain on noisy → clean pairs
Data compressionEncode to small latent, decode back
PretrainingPretrain encoder, then fine-tune
Python
# Anomaly detection: flag points with high reconstruction error
reconstructions = model(X_test)
errors = ((X_test - reconstructions) ** 2).mean(dim=1)
threshold = errors.quantile(0.95)
anomalies = errors > threshold
print(f"Anomalies detected: {anomalies.sum().item()}")

Exercise

Why can't a standard autoencoder be used directly as a generative model?

Need a hint?

Think about what happens when you sample a random point from the latent space.

Exercise

In a denoising autoencoder, what is the training objective?

Need a hint?

Think about what the input and target are.

Exercise

How can an autoencoder detect anomalous data?

Need a hint?

Think about what happens when an autoencoder sees unfamiliar data.

Common mistakes

  1. Confusing reconstruction with generation: an autoencoder can reconstruct inputs well but interpolating in latent space produces incoherent outputs because the latent space has "holes" (unenforced structure).
  2. Overfitting the bottleneck: a bottleneck that is too wide memorises rather than compresses; test reconstruction quality to detect this.
  3. Using autoencoders as generative models: standard autoencoders are not generative; they lack a prior over the latent space. Use a VAE if you need to sample.

Key takeaways

  • An autoencoder learns to compress (encoder) and reconstruct (decoder) data.
  • The bottleneck forces a useful low-dimensional code.
  • Variants: undercomplete, sparse, denoising, convolutional.
  • A plain autoencoder isn't generative — its latent space has gaps.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelated
AutoencodersAttention & TransformersPrincipal Component Analy…What Are Generative Model…Variational AutoencodersAnomaly & Outlier Detecti…