Design a Text-to-Image Generation Service
Track
Generative AI System Design
Inspired by
Midjourney / Stability AI
Scale
millions of images/day · GPU-bound · seconds per image
Difficulty
Intermediate
"Type a prompt, get an image" is the flagship generative-AI system-design problem. Unlike a classifier, the output is a sample from a learned distribution, so the challenges are generation quality, GPU cost (the model runs a heavy denoising loop), and safety (it can generate anything). Most of the pipeline is an instance of the ML loop (train a generative model), with a serving stage that dominates cost.
1. Clarify the problem
- Goal: given a text prompt, generate high-quality, prompt-faithful images, fast and affordably, at product scale.
- Quality bar: aesthetic quality and prompt adherence (the image matches what was asked) — two axes that trade off.
- Product surface: resolution, styles, editing, number of variants; safety is non-negotiable (no CSAM, no nonconsensual imagery, IP concerns).
- Scope: the generation service (model + serving + safety). Training the base model is in scope conceptually; in practice you'd fine-tune an open base like SDXL rather than train from scratch.
Interview signal
Name the two quality axes early — aesthetics vs. prompt faithfulness — and that the system is GPU-bound at inference. Those two facts drive the model, serving, and cost story.
Think firstStable Diffusion 1.x runs an ~860M-parameter UNet ~50 times per image. Walk the arithmetic: roughly how long is one image on a modern datacenter GPU, how many images per GPU-hour — and what fleet does 'millions of images/day' imply?Commit to an answer, then revealHide
Each denoising step is a full forward pass, so time ≈ steps × per-step cost — a few seconds per image at 50 steps, i.e. on the order of 1–2K images per GPU-hour. Millions of images a day therefore means a fleet of hundreds of GPUs running hot. And because cost is linear in step count, halving steps halves the fleet, which is why step reduction, not model shrinking, is the first cost lever every image product pulls.
2. Requirements & scale
| Dimension | Target | Consequence |
|---|---|---|
| Latency | Seconds per image | Fewer denoising steps; distilled/accelerated samplers |
| Cost | GPU is the dominant cost | Batching, step reduction, quantization, latent-space diffusion |
| Quality | Aesthetic + prompt-faithful | Strong text encoder + guidance + fine-tuning |
| Safety | Block harmful/IP-infringing output | Input + output filters, dataset curation |
Back-of-envelope: a diffusion model runs N denoising steps, each a full UNet/transformer forward pass on a GPU. Total cost is , so the whole cost story is "how do we cut , shrink each step, and pack GPUs", which is why latent diffusion (denoise a compressed representation) and step-distilled samplers matter so much.
3. GenAI problem framing
Conditional generative modeling. The dominant family is the latent diffusion model: an autoencoder compresses images into a latent space and a diffusion model learns to denoise latents conditioned on a text embedding (Rombach et al., 2022). At sampling time, classifier-free guidance (CFG) extrapolates away from the unconditional prediction,
trading diversity for prompt adherence as the guidance scale (typically 5–8) rises (Ho & Salimans, 2022).
What was the pre-generative baseline? Retrieval (search a stock library) or GANs — GANs are fast but unstable and mode-collapse-prone; diffusion won on quality and controllability.
Think firstFor a generative model, what does the training data determine beyond output quality?Commit to an answer, then revealHide
Everything the model can and can't be made to do. Curation — dedup, aesthetic filtering, NSFW/CSAM removal, caption quality — is simultaneously the quality lever, the first safety layer (a model can't easily generate what it never learned), and the legal exposure (training-data provenance and IP). For generative systems, dataset curation isn't preprocessing; it's product design.
4. Data & training (spine: data)
- Data: web-scale image–text pairs — LAION-5B is the canonical open corpus: 5.85B CLIP-filtered pairs, shipped with NSFW, watermark, and toxicity detection scores precisely so trainers can filter (Schuhmann et al., 2022). Curation is the product: dedup, aesthetic filtering, CSAM removal, and caption quality gate what the model can do, and what it can't be made to do.
- IP/licensing: training-data provenance is a legal and ethical minefield; document sources and honor opt-outs (Stability and others now run artist opt-out programs).
- Fine-tuning: start from an open base, then fine-tune / LoRA for style, brand, or safety; aesthetic fine-tuning on a small curated set moves human preference more than architecture changes.
Think firstStable Diffusion's autoencoder maps a 512×512×3 image to a 64×64×4 latent. Compute the reduction factor — and name what you risk losing by denoising there instead of in pixel space.Commit to an answer, then revealHide
786K values down to 16K — a ~48× reduction that every one of the ~50 denoising passes enjoys. That, more than any sampler trick, is what made text-to-image affordable. The price: final quality is capped by the VAE decoder, so high-frequency detail the 8×-downsampled latent can't carry — legible text, faces, hands — becomes the signature failure mode, and decoder upgrades ship as quality wins on their own.
5. Model & generation (spine: hypothesis-space + objective)
- Text encoder: prompt faithfulness lives or dies here. Imagen showed that scaling a frozen T5-XXL text encoder improved both fidelity and image–text alignment more than scaling the diffusion model itself (Saharia et al., 2022); SDXL runs two text encoders (CLIP ViT-L + OpenCLIP ViT-bigG) feeding a 2.6B-parameter UNet (Podell et al., 2023).
- Denoiser: a UNet or diffusion transformer over latents, trained with the denoising (noise-prediction) objective.
- Sampling: DDIM/DPM-Solver at 20–50 steps + CFG for adherence; step distillation collapses this — adversarial diffusion distillation (SDXL Turbo) samples in 1–4 steps at near-teacher quality (Sauer et al., 2023).
Diffusion Process — Forward & Reverse
Forward: watch structure dissolve into Gaussian noise over 20 steps. Reverse: watch a denoising network reconstruct structure from noise. Toggle the β schedule to compare linear vs cosine noise addition.
Diffusion learns to reverse a gradual noising process — generation is running that reverse chain from pure noise, guided by the text embedding.
Think firstOne image costs 20–50 denoising steps, each a full forward pass on a GPU. Enumerate the cost levers before reading on — and estimate what each is worth.Commit to an answer, then revealHide
Attack each factor of the product: fewer steps (distillation gets 50 → 1–4, a ~10–25× win), cheaper steps (the ~48× latent compression, plus quantizing the denoiser), and better GPU packing (batch concurrent requests, autoscale by queue depth). Caching — identical prompts, reusable text-encoder embeddings, seeds for variants — is the bonus lever product teams forget. Note the levers multiply, and step count dominates.
6. Serving & scaling
The crux. Per request: encode prompt → run N denoising steps on GPU → decode latent → safety-check → return. Optimize by:
- Fewer steps (distilled samplers: 50 → 1–4), latent-space diffusion (~48× smaller tensors per step), quantization (fp16/int8 denoiser), and batching concurrent requests to saturate GPUs.
- Autoscaling GPU pools by queue depth; a job queue with priority tiers (interactive draft vs. batch upscale) — Midjourney's "fast/relax" modes are exactly this split.
- Caching text-encoder embeddings and identical prompt+seed results.
Think firstFID improved on the new checkpoint. Ship it? What do FID and CLIPScore each actually measure — and what do both miss?Commit to an answer, then revealHide
FID measures distributional realism (do generated images statistically resemble real ones); CLIPScore measures prompt adherence. Neither measures what users buy: aesthetic taste. Both can improve while humans prefer the old model, so automated metrics screen checkpoints, and human preference A/B makes the ship call.
7. Evaluation (spine: evaluation)
- Automated: FID (distributional realism) and CLIPScore (prompt adherence) — necessary but weakly correlated with human taste; run them on a fixed prompt suite spanning composition, counting, text rendering, and style.
- Human preference: side-by-side A/B on aesthetics + adherence is the real judge; collect preference data continuously from user picks and upscales.
- Safety evals: red-team prompt suites for harmful/IP-infringing generations, tracked per policy category.
Think firstA generation comes back with a stock-photo watermark ghosted into the corner. The model has no retrieval component. What happened — and which training-time decision most reduces it?Commit to an answer, then revealHide
The model memorized. Diffusion models trained on billions of web images can emit near-copies of training examples — Carlini et al. extracted over a thousand training images from Stable Diffusion with a generate-and-filter attack, and found heavily duplicated training images are by far the likeliest to regurgitate. So deduplication is the highest-leverage mitigation, and extraction-style audits belong in your eval suite. This is an IP and privacy exposure, not a curiosity.
8. Safety & guardrails
- Input filtering: classify prompts and block clearly disallowed requests before spending a generation.
- Output filtering: a classifier on generated images (CSAM, NSFW, violence, likeness of real people) before returning. DALL·E 3 ships exactly this layered stack — external red-teaming plus prompt- and image-level classifiers with category-specific handling (OpenAI, 2023).
- Provenance: watermark outputs (visible/invisible + C2PA signed metadata) so AI images are identifiable downstream.
- Memorization/IP: dedup training data and monitor for training-image regurgitation (Carlini et al., 2023).
9. Feedback & iteration (spine: feedback)
- Preference data from user picks/upscales feeds fine-tuning (RLHF-style alignment for aesthetics/safety) — the pick-one-of-four UI is a preference labeler in disguise.
- Drift: prompt trends shift; monitor rejection/safety-filter rates and refresh models.
10. Tradeoffs & alternatives
| Choice | Alternative | Why |
|---|---|---|
| Latent diffusion | Pixel-space diffusion | ~48× smaller tensors per denoising step |
| Diffusion | GANs | Better quality/diversity/control; GANs are faster but unstable |
| Distilled 1–4-step sampling | Full 50-step sampling | ~10–25× less GPU per image at slight quality loss |
| Fine-tune open base | Train from scratch | Orders of magnitude cheaper; base models are strong |
11. Interviewer follow-ups
Images look great but ignore half the prompt. How do you improve adherence?
Prompt faithfulness is driven by the text encoder, the captions, and guidance. Raise the CFG scale (more adherence, less diversity, but too high causes oversaturated artifacts), use a stronger/larger text encoder (the Imagen result), and fine-tune on high-quality caption–image pairs. Measure with CLIPScore + human adherence ratings, not just FID (which only captures realism).
Generation takes 8 seconds and GPU costs are killing us. What do you do?
Cut denoising steps with a distilled sampler (adversarial or consistency distillation reaches 1–4 steps), stay in latent space, quantize the denoiser, and batch requests to saturate GPUs. Each step is a full forward pass, so fewer steps × smaller tensors × better packing is the whole cost lever — and step count is the dominant factor.
A user prompts for a real celebrity in a compromising scene. How is that blocked?
Layered: an input filter rejects the prompt class, an output classifier flags disallowed likeness/NSFW before returning, and training data + fine-tuning reduce the model's ability to produce it. No single layer is sufficient — safety is defense-in-depth, and provenance watermarking aids downstream accountability.
Key takeaways
- Text-to-image is conditional generative modeling (latent diffusion + CFG) — the axes are aesthetics vs. prompt faithfulness.
- The system is GPU-bound: fewer steps (distillation to 1–4), latent-space diffusion (~48× compression), quantization, and batching are the cost levers — and they multiply.
- Data curation defines capability, safety, and memorization risk; layered input/output filtering + watermarking are mandatory, not optional.
References
- High-Resolution Image Synthesis with Latent Diffusion Models: Rombach et al., CVPR 2022; the latent-diffusion architecture and its compression economics.
- Classifier-Free Diffusion Guidance: Ho & Salimans, 2022; the CFG sampling rule and its fidelity–diversity tradeoff.
- Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding: Saharia et al., NeurIPS 2022 (Imagen); scaling the frozen T5 text encoder beats scaling the diffusion model.
- SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis: Podell et al., Stability AI, 2023; the 2.6B UNet, dual text encoders, and refiner design.
- Adversarial Diffusion Distillation: Sauer et al., Stability AI, 2023 (SDXL Turbo); 1–4-step sampling via score + adversarial distillation.
- LAION-5B: An open large-scale dataset for training next generation image-text models: Schuhmann et al., NeurIPS 2022; 5.85B CLIP-filtered pairs with NSFW/watermark scores, the canonical training corpus.
- Extracting Training Data from Diffusion Models: Carlini et al., USENIX Security 2023; memorization/regurgitation risk and the dedup mitigation.
- DALL·E 3 System Card: OpenAI, 2023; a deployed example of layered red-teaming, prompt filtering, and output classification.