ViT & Modern Generative AI
The past five years brought a wave of architectural innovations that pushed generative models from toy demos to photorealistic, billion-parameter systems. This lesson covers five of the most influential ideas: the Vision Transformer, Classifier-Free Guidance, StyleGAN, CycleGAN, and the Stable Diffusion latent pipeline.
1. Vision Transformer (ViT)
Transformers dominated NLP. The natural question was: can the same architecture handle images? The answer turned out to be yes — if you have enough data.
Patch embedding
A raw image can't be fed as a single sequence token — it's too large. ViT's solution: chop the image into non-overlapping square patches, flatten each patch, and project it linearly into a token embedding.
Worked example: Take a 224×224 RGB image with patch size .
- Number of patches:
- Pixels per patch: raw values
- Each patch is projected by a learned weight matrix to a -dimensional embedding (ViT-B uses )
So the entire image becomes a sequence of 196 tokens, each of dimension 768. That's an input matrix of shape — a standard Transformer can process it directly.
import numpy as np
def patch_embed(image, patch_size, d_model):
"""
image: (H, W, C) numpy array
Returns: (num_patches, d_model) token matrix
"""
H, W, C = image.shape
P = patch_size
assert H % P == 0 and W % P == 0
# Extract patches
patches = image.reshape(H // P, P, W // P, P, C)
patches = patches.transpose(0, 2, 1, 3, 4) # (n_h, n_w, P, P, C)
n_h, n_w = H // P, W // P
patches = patches.reshape(n_h * n_w, P * P * C) # (196, 768)
# Linear projection (random init for illustration)
rng = np.random.default_rng(0)
W_E = rng.normal(0, 0.02, (P * P * C, d_model))
return patches @ W_E # (196, 768)
image = np.random.randn(224, 224, 3)
tokens = patch_embed(image, patch_size=16, d_model=768)
print(f"Token matrix shape: {tokens.shape}") # (196, 768)
[CLS] token and positional embeddings
Two additions are prepended/added before passing tokens to the Transformer encoder:
- [CLS] token: a learned vector prepended to the sequence. After all encoder layers, only this token's representation is used for classification (it aggregates global information via attention).
- Positional embeddings: learnable 1D vectors of shape added element-wise to the 196 patch tokens plus the CLS token. Unlike the sinusoidal embeddings in NLP Transformers, ViT's positional embeddings are fully learned from data.
where is the patch projection and is the learnable positional embedding table.
Encoder and classification head
The sequence passes through standard Transformer encoder blocks (self-attention + MLP, with LayerNorm). There is no causal mask — every patch can attend to every other patch bidirectionally.
Finally, the [CLS] token's output vector goes into a small MLP head:
where is the CLS position at the final layer .
Scale comparison
| Variant | Patch size | Layers | Heads | Params | |
|---|---|---|---|---|---|
| ViT-B/16 | 16×16 | 12 | 12 | 768 | 86 M |
| ViT-L/16 | 16×16 | 24 | 16 | 1024 | 307 M |
| ViT-H/14 | 14×14 | 32 | 16 | 1280 | 632 M |
Data requirements
CNNs bake in two inductive biases: locality (nearby pixels matter) and translation equivariance (a cat anywhere in the image is still a cat). ViT has neither — it must learn spatial relationships from scratch. On small datasets like ImageNet-1k, CNNs outperform ViT. But trained on JFT-300M (300 million images), ViT matches or beats the best CNNs. Data is the inductive bias at scale.
2. Classifier-Free Guidance (CFG)
Conditional generation (e.g., "generate a golden retriever") requires the model to follow a text or class prompt faithfully while still producing diverse, high-quality samples. Classifier-Free Guidance achieves this without an external classifier.
Training
Train the model jointly on two modes by randomly dropping the conditioning signal during training:
- With probability : condition on the real label/text
- With probability : replace with a null token (the "unconditional" case)
The model therefore learns both and with one set of weights.
Inference
At generation time, run the model twice per denoising step (once with and once without the condition), then interpolate:
where is the guidance scale:
| Effect | |
|---|---|
| 0 | No guidance: the plain conditional output (the form gives the conditional prediction at , never the unconditional one) |
| 1 | Moderate guidance: the prompt direction is amplified once |
| 7.5 | Strong guidance: high prompt fidelity, less diversity (≈ Stable Diffusion's default; see convention note below) |
| 15+ | Very strong: may over-saturate colors and lose realism |
Convention note: libraries like
diffusersexposeguidance_scalein the form , which equals the formula above with . So "Stable Diffusion's default 7.5" is , i.e. here — same idea, off-by-one parameterization.
Intuition: The term is the direction in noise-space that leads toward images matching the condition . Amplifying it by steers sampling more aggressively toward the prompt.
def cfg_output(cond_output, uncond_output, guidance_scale):
"""Combine conditional and unconditional model outputs with CFG."""
w = guidance_scale
return (1 + w) * cond_output - w * uncond_output
import numpy as np
cond = np.array([0.8, -0.2, 0.5])
uncond = np.array([0.1, 0.1, 0.1])
print(cfg_output(cond, uncond, w=0)) # → [ 0.8 -0.2 0.5] (conditional, no guidance)
print(cfg_output(cond, uncond, w=7.5)) # → [ 6.05 -2.45 3.5] (strong pull toward cond)
CFG costs 2× compute
Each denoising step requires two forward passes. This doubles inference cost. Some modern systems (e.g., SDXL Turbo, Stable Diffusion 3) use distillation to bake guidance into a single forward pass.
3. StyleGAN
Standard GANs feed a latent vector directly to the generator. This couples all aspects of style into a single vector, making it hard to control individual attributes like pose, color, and fine texture independently. StyleGAN's insight: disentangle style from structure with a mapping network and adaptive instance normalization.
Mapping network
Instead of feeding directly to the generator, first pass it through a small 8-layer MLP:
The -space is less entangled than -space because the mapping network can learn to spread features apart. A vector in is often replicated and injected at every scale of the generator.
AdaIN (Adaptive Instance Normalization)
At each generator layer, the feature map is normalized and then shifted/scaled by style vectors derived from :
where and are per-channel scale and bias derived from via a learned affine transform. Each scale receives a different projection of , allowing independent control at each resolution.
Progressive growing
Generation starts at 4×4 resolution, with new layers added progressively (4→8→16→32→64→…→1024) during training. This stabilizes training: the network first learns gross structure, then progressively refines detail.
What the W-space gives you: By linearly interpolating between two vectors injected at coarse vs. fine layers, you can mix styles — e.g., take the pose from one face and the color/texture from another.
4. CycleGAN
Paired image translation (photo → sketch where every photo has a matching sketch) is simple. CycleGAN handles the harder case: unpaired translation, where you only have a collection of domain-A images and a separate collection of domain-B images, with no correspondence.
Architecture
Two generator–discriminator pairs:
- : translates A → B; : tries to distinguish real B from generated B
- : translates B → A; : tries to distinguish real A from generated A
Cycle consistency loss
The key insight: if we translate an image from A to B and then back to A, we should recover the original:
This L1 cycle loss prevents the generators from freely hallucinating irrelevant content in the target domain — they must produce translations that can be inverted.
def cycle_consistency_loss(x, G_AB, G_BA):
"""Compute L1 cycle consistency loss for a batch x from domain A."""
x_in_B = G_AB(x) # translate A → B
x_reconstructed = G_BA(x_in_B) # translate back B → A
return np.mean(np.abs(x_reconstructed - x))
Total loss
where is the cycle consistency weight. The GAN losses push generators to produce realistic samples in the target domain; the cycle loss preserves content.
Classic applications: horse → zebra, photo → Monet painting, summer → winter, day → night.
Why L1 and not L2?
L1 loss produces sharper results than L2 (which tends to blur) because L1 penalizes all deviations equally rather than heavily penalizing outliers. This matters for preserving edge structure across domains.
5. Stable Diffusion: Latent Diffusion
Pixel-space diffusion (DALL-E 2, original DDPM) runs the denoising U-Net on full-resolution images. A 512×512×3 image has 786,432 values — every denoising step processes all of them. Stable Diffusion moves the diffusion process into a compressed latent space, making it orders of magnitude cheaper.
The three-component pipeline
Step 1 — VAE encoder (compress):
The VAE compresses the image by a factor of 8 in each spatial dimension, reducing the tensor from 786 K values to 16 K values — a 48× reduction (). The VAE is pre-trained separately and frozen during diffusion training.
Step 2 — Latent diffusion (denoise):
The U-Net denoiser now operates in -space. It is conditioned on text via cross-attention at every U-Net block:
- Text is encoded by a frozen CLIP text encoder:
- Each U-Net block has a cross-attention layer where queries come from image features and keys/values come from
Inference loop (DDIM, 20–50 steps):
z_T ~ N(0, I) # start from latent noise
for t in [T, T-1, ..., 1]:
eps_hat = U-Net(z_t, t, c) # predict noise (conditioned on text c)
z_{t-1} = DDIM_step(z_t, eps_hat, t)
x = VAE_decoder(z_0) # decode final latent to pixels
Step 3 — VAE decoder (decompress):
Tensor shapes through the pipeline
# Illustrative shapes — no real model needed
import numpy as np
prompt_tokens = (77,) # CLIP max token length
text_emb = (77, 768) # CLIP text embedding
latent_noise = (1, 4, 64, 64) # z_T ~ N(0,I) [B, C, H, W]
unet_output = (1, 4, 64, 64) # predicted noise, same shape as latent
final_latent = (1, 4, 64, 64) # z_0 after DDIM sampling
decoded_image = (1, 3, 512, 512)# VAE decoder output
print(f"Text embedding: {text_emb}")
print(f"Latent noise: {latent_noise}")
print(f"U-Net output: {unet_output}")
print(f"Decoded image: {decoded_image}")
print(f"Compression ratio: {512*512*3 / (64*64*4):.0f}×") # → 48×
Why latent space works
The VAE is trained with a perceptual loss — it learns to preserve visually important structure (edges, textures) while discarding imperceptible high-frequency noise. The diffusion model only needs to learn the distribution of perceptually meaningful latents, not raw pixel values. This separation of concerns is the key architectural insight of Stable Diffusion.
Summary of the pipeline
Encode (VAE) → Denoise (U-Net conditioned by CLIP) → Decode (VAE). The middle step runs in a 48× smaller space, so each denoising step is ~48× cheaper than pixel-space diffusion at the same resolution.
GAN vs VAE vs Diffusion: Side-by-side
| Property | GAN | VAE | Diffusion |
|---|---|---|---|
| Sample quality | Very good | Good | Best |
| Sample diversity | Risk of mode collapse | Good | Excellent |
| Training stability | Unstable (adversarial) | Stable | Stable |
| Inference speed | Fast (1 forward pass) | Fast (1 forward pass) | Slow (20–1000 steps) |
| Likelihood estimate | None | Lower bound (ELBO) | Lower bound |
| Conditioning | Conditional GAN (tricky) | Simple (concat/AdaIN) | CFG (flexible, costly) |
| Key failure mode | Mode collapse / training instability | Posterior collapse / blurry outputs | Slow generation |
| Flagship system | StyleGAN3 | Stable Diffusion VAE | Stable Diffusion, DALL-E 3 |
Exercise
A Vision Transformer (ViT) processes a 224×224 image with patch size P=16. How many patches (tokens) does it create, and what is the dimension of each flattened patch?
Need a hint?
Divide each spatial dimension by P to get patch grid size; flatten the patch to a vector.
Exercise
Classifier-Free Guidance with scale w=7.5 combines conditioned and unconditioned predictions as: output = (1+w)·model(x,c) − w·model(x,∅). What happens as w increases?
Need a hint?
Think about what happens to the unconditioned component as w grows.
Exercise
CycleGAN uses a cycle consistency loss. If G_AB translates domain A→B and G_BA translates B→A, what does the cycle consistency loss enforce?
Need a hint?
What should happen if you translate an image there and back?
Common mistakes
- Feeding raw pixels to ViT without enough data: ViT has no locality bias, so it needs large datasets. Use a pre-trained checkpoint for fine-tuning; don't train ViT-B from scratch on ImageNet-1k alone.
- Setting CFG guidance too high: causes over-saturation and artifact-heavy images. The sweet spot is typically for photorealistic prompts.
- Confusing Z-space and W-space in StyleGAN: Only -space interpolations give semantically smooth transitions. Interpolating in passes through the non-linear mapping network and produces inconsistent results.
- Forgetting both cycle directions in CycleGAN: The loss must enforce AND . Omitting either direction lets one generator collapse.
- Treating Stable Diffusion as pixel-space diffusion: The U-Net operates on 64×64 latents, not 512×512 pixels. Interventions like inpainting or ControlNet operate in latent space and must be decoded only at the end.
Key takeaways
- ViT treats an image as a sequence of patch tokens. With enough data it matches CNN performance, but inductive biases matter at small data scales.
- Classifier-Free Guidance trains one model for both conditional and unconditional generation; at inference it amplifies the direction toward the condition by guidance scale .
- StyleGAN uses a mapping network () and AdaIN to disentangle style at each scale, enabling fine-grained image attribute control.
- CycleGAN enables unpaired image translation via cycle consistency: translating there and back must recover the original.
- Stable Diffusion runs diffusion in a VAE latent space (~48× smaller than pixel space), with CLIP-conditioned cross-attention in the U-Net, making high-resolution synthesis tractable.
Related concepts
- Diffusion Models: the denoising foundation Stable Diffusion builds on
- Generative Adversarial Networks: the adversarial training framework StyleGAN and CycleGAN use
- Variational Autoencoders: the VAE component inside Stable Diffusion
- Attention & Transformers: the self-attention mechanism ViT inherits