Design a Text-to-Image Generation Service
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Design a Text-to-Image Generation Service

Track

Generative AI System Design

Inspired by

Midjourney / Stability AI

Scale

millions of images/day · GPU-bound · seconds per image

Difficulty

Intermediate

"Type a prompt, get an image" is the flagship generative-AI system-design problem. Unlike a classifier, the output is a sample from a learned distribution, so the challenges are generation quality, GPU cost (the model runs a heavy denoising loop), and safety (it can generate anything). Most of the pipeline is an instance of the ML loop (train a generative model), with a serving stage that dominates cost.

1. Clarify the problem

  • Goal: given a text prompt, generate high-quality, prompt-faithful images, fast and affordably, at product scale.
  • Quality bar: aesthetic quality and prompt adherence (the image matches what was asked) — two axes that trade off.
  • Product surface: resolution, styles, editing, number of variants; safety is non-negotiable (no CSAM, no nonconsensual imagery, IP concerns).
  • Scope: the generation service (model + serving + safety). Training the base model is in scope conceptually; in practice you'd fine-tune an open base like SDXL rather than train from scratch.

Interview signal

Name the two quality axes early — aesthetics vs. prompt faithfulness — and that the system is GPU-bound at inference. Those two facts drive the model, serving, and cost story.

Think firstStable Diffusion 1.x runs an ~860M-parameter UNet ~50 times per image. Walk the arithmetic: roughly how long is one image on a modern datacenter GPU, how many images per GPU-hour — and what fleet does 'millions of images/day' imply?Commit to an answer, then reveal

Each denoising step is a full forward pass, so time ≈ steps × per-step cost — a few seconds per image at 50 steps, i.e. on the order of 1–2K images per GPU-hour. Millions of images a day therefore means a fleet of hundreds of GPUs running hot. And because cost is linear in step count, halving steps halves the fleet, which is why step reduction, not model shrinking, is the first cost lever every image product pulls.

2. Requirements & scale

DimensionTargetConsequence
LatencySeconds per imageFewer denoising steps; distilled/accelerated samplers
CostGPU is the dominant costBatching, step reduction, quantization, latent-space diffusion
QualityAesthetic + prompt-faithfulStrong text encoder + guidance + fine-tuning
SafetyBlock harmful/IP-infringing outputInput + output filters, dataset curation

Back-of-envelope: a diffusion model runs N denoising steps, each a full UNet/transformer forward pass on a GPU. Total cost is N×cost(step)N \times \text{cost}(\text{step}), so the whole cost story is "how do we cut NN, shrink each step, and pack GPUs", which is why latent diffusion (denoise a compressed representation) and step-distilled samplers matter so much.

3. GenAI problem framing

Conditional generative modeling. The dominant family is the latent diffusion model: an autoencoder compresses images into a latent space and a diffusion model learns to denoise latents conditioned on a text embedding (Rombach et al., 2022). At sampling time, classifier-free guidance (CFG) extrapolates away from the unconditional prediction,

ϵ^=ϵuncond+w (ϵcond−ϵuncond),\hat{\epsilon} = \epsilon_{\text{uncond}} + w\,(\epsilon_{\text{cond}} - \epsilon_{\text{uncond}}),

trading diversity for prompt adherence as the guidance scale ww (typically 5–8) rises (Ho & Salimans, 2022).

What was the pre-generative baseline? Retrieval (search a stock library) or GANs — GANs are fast but unstable and mode-collapse-prone; diffusion won on quality and controllability.

Think firstFor a generative model, what does the training data determine beyond output quality?Commit to an answer, then reveal

Everything the model can and can't be made to do. Curation — dedup, aesthetic filtering, NSFW/CSAM removal, caption quality — is simultaneously the quality lever, the first safety layer (a model can't easily generate what it never learned), and the legal exposure (training-data provenance and IP). For generative systems, dataset curation isn't preprocessing; it's product design.

4. Data & training (spine: data)

  • Data: web-scale image–text pairs — LAION-5B is the canonical open corpus: 5.85B CLIP-filtered pairs, shipped with NSFW, watermark, and toxicity detection scores precisely so trainers can filter (Schuhmann et al., 2022). Curation is the product: dedup, aesthetic filtering, CSAM removal, and caption quality gate what the model can do, and what it can't be made to do.
  • IP/licensing: training-data provenance is a legal and ethical minefield; document sources and honor opt-outs (Stability and others now run artist opt-out programs).
  • Fine-tuning: start from an open base, then fine-tune / LoRA for style, brand, or safety; aesthetic fine-tuning on a small curated set moves human preference more than architecture changes.
Deep dive
DDIM — fast deterministic diffusion sampling
Includes runnable notebook
Think firstStable Diffusion's autoencoder maps a 512×512×3 image to a 64×64×4 latent. Compute the reduction factor — and name what you risk losing by denoising there instead of in pixel space.Commit to an answer, then reveal

786K values down to 16K — a ~48× reduction that every one of the ~50 denoising passes enjoys. That, more than any sampler trick, is what made text-to-image affordable. The price: final quality is capped by the VAE decoder, so high-frequency detail the 8×-downsampled latent can't carry — legible text, faces, hands — becomes the signature failure mode, and decoder upgrades ship as quality wins on their own.

5. Model & generation (spine: hypothesis-space + objective)

  • Text encoder: prompt faithfulness lives or dies here. Imagen showed that scaling a frozen T5-XXL text encoder improved both fidelity and image–text alignment more than scaling the diffusion model itself (Saharia et al., 2022); SDXL runs two text encoders (CLIP ViT-L + OpenCLIP ViT-bigG) feeding a 2.6B-parameter UNet (Podell et al., 2023).
  • Denoiser: a UNet or diffusion transformer over latents, trained with the denoising (noise-prediction) objective.
  • Sampling: DDIM/DPM-Solver at 20–50 steps + CFG for adherence; step distillation collapses this — adversarial diffusion distillation (SDXL Turbo) samples in 1–4 steps at near-teacher quality (Sauer et al., 2023).

Diffusion Process — Forward & Reverse

modeForward
t0
β_t—
σ_t—

Forward: watch structure dissolve into Gaussian noise over 20 steps. Reverse: watch a denoising network reconstruct structure from noise. Toggle the β schedule to compare linear vs cosine noise addition.

Diffusion learns to reverse a gradual noising process — generation is running that reverse chain from pure noise, guided by the text embedding.

Think firstOne image costs 20–50 denoising steps, each a full forward pass on a GPU. Enumerate the cost levers before reading on — and estimate what each is worth.Commit to an answer, then reveal

Attack each factor of the product: fewer steps (distillation gets 50 → 1–4, a ~10–25× win), cheaper steps (the ~48× latent compression, plus quantizing the denoiser), and better GPU packing (batch concurrent requests, autoscale by queue depth). Caching — identical prompts, reusable text-encoder embeddings, seeds for variants — is the bonus lever product teams forget. Note the levers multiply, and step count dominates.

6. Serving & scaling

The crux. Per request: encode prompt → run N denoising steps on GPU → decode latent → safety-check → return. Optimize by:

  • Fewer steps (distilled samplers: 50 → 1–4), latent-space diffusion (~48× smaller tensors per step), quantization (fp16/int8 denoiser), and batching concurrent requests to saturate GPUs.
  • Autoscaling GPU pools by queue depth; a job queue with priority tiers (interactive draft vs. batch upscale) — Midjourney's "fast/relax" modes are exactly this split.
  • Caching text-encoder embeddings and identical prompt+seed results.
Think firstFID improved on the new checkpoint. Ship it? What do FID and CLIPScore each actually measure — and what do both miss?Commit to an answer, then reveal

FID measures distributional realism (do generated images statistically resemble real ones); CLIPScore measures prompt adherence. Neither measures what users buy: aesthetic taste. Both can improve while humans prefer the old model, so automated metrics screen checkpoints, and human preference A/B makes the ship call.

7. Evaluation (spine: evaluation)

  • Automated: FID (distributional realism) and CLIPScore (prompt adherence) — necessary but weakly correlated with human taste; run them on a fixed prompt suite spanning composition, counting, text rendering, and style.
  • Human preference: side-by-side A/B on aesthetics + adherence is the real judge; collect preference data continuously from user picks and upscales.
  • Safety evals: red-team prompt suites for harmful/IP-infringing generations, tracked per policy category.
Think firstA generation comes back with a stock-photo watermark ghosted into the corner. The model has no retrieval component. What happened — and which training-time decision most reduces it?Commit to an answer, then reveal

The model memorized. Diffusion models trained on billions of web images can emit near-copies of training examples — Carlini et al. extracted over a thousand training images from Stable Diffusion with a generate-and-filter attack, and found heavily duplicated training images are by far the likeliest to regurgitate. So deduplication is the highest-leverage mitigation, and extraction-style audits belong in your eval suite. This is an IP and privacy exposure, not a curiosity.

8. Safety & guardrails

  • Input filtering: classify prompts and block clearly disallowed requests before spending a generation.
  • Output filtering: a classifier on generated images (CSAM, NSFW, violence, likeness of real people) before returning. DALL·E 3 ships exactly this layered stack — external red-teaming plus prompt- and image-level classifiers with category-specific handling (OpenAI, 2023).
  • Provenance: watermark outputs (visible/invisible + C2PA signed metadata) so AI images are identifiable downstream.
  • Memorization/IP: dedup training data and monitor for training-image regurgitation (Carlini et al., 2023).

9. Feedback & iteration (spine: feedback)

  • Preference data from user picks/upscales feeds fine-tuning (RLHF-style alignment for aesthetics/safety) — the pick-one-of-four UI is a preference labeler in disguise.
  • Drift: prompt trends shift; monitor rejection/safety-filter rates and refresh models.

10. Tradeoffs & alternatives

ChoiceAlternativeWhy
Latent diffusionPixel-space diffusion~48× smaller tensors per denoising step
DiffusionGANsBetter quality/diversity/control; GANs are faster but unstable
Distilled 1–4-step samplingFull 50-step sampling~10–25× less GPU per image at slight quality loss
Fine-tune open baseTrain from scratchOrders of magnitude cheaper; base models are strong

11. Interviewer follow-ups

Images look great but ignore half the prompt. How do you improve adherence?

Prompt faithfulness is driven by the text encoder, the captions, and guidance. Raise the CFG scale (more adherence, less diversity, but too high causes oversaturated artifacts), use a stronger/larger text encoder (the Imagen result), and fine-tune on high-quality caption–image pairs. Measure with CLIPScore + human adherence ratings, not just FID (which only captures realism).

Generation takes 8 seconds and GPU costs are killing us. What do you do?

Cut denoising steps with a distilled sampler (adversarial or consistency distillation reaches 1–4 steps), stay in latent space, quantize the denoiser, and batch requests to saturate GPUs. Each step is a full forward pass, so fewer steps × smaller tensors × better packing is the whole cost lever — and step count is the dominant factor.

A user prompts for a real celebrity in a compromising scene. How is that blocked?

Layered: an input filter rejects the prompt class, an output classifier flags disallowed likeness/NSFW before returning, and training data + fine-tuning reduce the model's ability to produce it. No single layer is sufficient — safety is defense-in-depth, and provenance watermarking aids downstream accountability.

Key takeaways

  • Text-to-image is conditional generative modeling (latent diffusion + CFG) — the axes are aesthetics vs. prompt faithfulness.
  • The system is GPU-bound: fewer steps (distillation to 1–4), latent-space diffusion (~48× compression), quantization, and batching are the cost levers — and they multiply.
  • Data curation defines capability, safety, and memorization risk; layered input/output filtering + watermarking are mandatory, not optional.

References