Design a GenAI Safety & Output Moderation System
Track
Generative AI System Design
Inspired by
cross-industry
Scale
every generation · low added latency · adversarial · policy-driven
Difficulty
Advanced
Every generative product needs a safety layer wrapping the model — the system that decides what prompts are allowed, what outputs may be returned, and how to mark generated content. It's a cross-cutting generative-AI system-design problem: a defense-in-depth pipeline against an adversarial user, added to every generation with minimal latency. It reads through the agentic loop's guardrails and evaluation stages.
1. Clarify the problem
- Goal: prevent a generative model from producing disallowed content (harmful, illegal, privacy-violating) and from being manipulated (jailbreaks, prompt injection), while marking outputs as AI-generated — without wrecking latency or over-blocking benign use.
- Adversarial by nature: users actively try to bypass it, so it's an arms race.
- Policy-driven: "allowed" is defined by a policy taxonomy that evolves; thresholds differ by category (self-harm vs. mild profanity).
- Scope: the safety layer around any generation model (text/image/audio/video).
Defense-in-depth, not one classifier
No single filter is robust against a determined adversary. The design is layers — input moderation, a hardened system policy, output moderation, and model-level alignment — each catching what the others miss.
Think firstBefore a single classifier can be trained: what does 'unsafe' mean, operationally? Name the artifacts that must exist first.Commit to an answer, then revealHide
A written policy taxonomy — categories and subcategories (sexual content, hate, violence, self-harm, harassment, …) with definitions, edge cases, and severity tiers — plus labeling instructions precise enough that two annotators agree, and per-category thresholds because error costs differ wildly. OpenAI's moderation team found the taxonomy and labeling design were the hard part, ahead of modeling, and that rare-but-catastrophic categories need active learning to even find training examples. Policy is the spec; the classifier is just its compiler.
2. Requirements & constraints
| Dimension | Target | Consequence |
|---|---|---|
| Coverage | Block disallowed in + out | Input + output classifiers per modality |
| Latency | Minimal added latency | Small fast classifiers; parallel checks |
| Robustness | Resist jailbreaks/injection | Adversarial training; layered defenses |
| Precision | Don't over-block benign use | Calibrated, per-category thresholds |
| Auditability | Explain + appeal decisions | Logging, reason codes, human review |
Think firstYour adversary gets unlimited retries against a static defense. What layers would you stack — and what does each catch that the others miss?Commit to an answer, then revealHide
Input moderation catches obviously disallowed requests before you pay for a generation; the hardened system policy resists manipulation (jailbreaks, injection) that slips past input checks; output moderation catches disallowed generations however they were induced; and model-level alignment makes the model refuse even when every wrapper fails. Each layer's misses are a different population — that's why stacking works and any single layer loses the arms race.
3. The layered pipeline (spine: guardrails)
Around each generation:
- Input moderation: classify the prompt against the taxonomy; block clearly disallowed requests before spending a generation (Markov et al., 2022).
- Hardened system policy: the model's instructions/policy that user (and retrieved) text cannot override — the core prompt-injection / jailbreak defense.
- Output moderation: classify the generation (text toxicity, image NSFW/ likeness, etc.) before returning; block or regenerate.
- Provenance: watermark and tag outputs (C2PA signed metadata, invisible marks) so generated content is identifiable downstream.
Think firstWhat actually is a 'moderation classifier' in 2025 — a keyword list, a fine-tuned encoder, or an LLM? What did Llama Guard's design change?Commit to an answer, then revealHide
The stack moved from blocklists → fine-tuned encoders → safeguard LLMs. Llama Guard is a Llama-2-7B instruction-tuned to classify both prompts and responses against a taxonomy that is written into its prompt, so you can swap or extend the policy zero-shot without retraining, and get per-category verdicts with a rationale. The tradeoff: an LLM judge costs more per call than an encoder, which is why production stacks distill or cascade (cheap encoder first, LLM safeguard on the uncertain slice).
4. The classifiers & alignment
- Moderation classifiers: fast per-category models — fine-tuned encoders, distilled LLMs, or safeguard LLMs like Llama Guard with prompt-swappable taxonomies (Inan et al., 2023) — for input and output, small enough to add little latency.
- Model-level alignment: RLHF/DPO with a reward/safety model so the base model refuses appropriately on its own (the deepest layer). Jailbreaks work by exploiting competing objectives (helpfulness vs. safety) and mismatched generalization (attacks phrased outside the safety-training distribution) (Wei et al., 2023) — name the failure mode before picking the fix.
- Human-in-the-loop: uncertain cases and appeals route to reviewers, whose decisions become training labels.
Reward model: pairwise preference → reward margin
Two candidate responses to one prompt, the chosen one (✓) and the rejected one (✗). Each gets a scalar reward from the reward model. The Bradley–Terry objective trains the model so the chosen reward sits above the rejected — the wider the margin, the higher the win probability σ(r_c − r_r) and the lower the loss −log σ(r_c − r_r).
A reward/safety model scores outputs for policy compliance — used both to align the base model (RLHF) and, distilled, as an output classifier at serving time.
Think firstSafety recall hits 99.9%. Is the system good? What's the other side of the ledger — and what would a production-grade operating point look like, in numbers?Commit to an answer, then revealHide
Over-refusal. Recall alone is trivially maxed by blocking everything — and a safety layer that refuses benign medical questions or legitimate creative work silently kills the product. The real objective is a calibrated operating point per category, and it's achievable: Anthropic's Constitutional Classifiers survived 3,000+ hours of red-teaming with no universal jailbreak found, at an absolute +0.38% refusal increase on production traffic. That pair of numbers — robustness gained vs. benign traffic lost — is the honest way to report any safety system.
5. Evaluation (spine: evaluation)
- Safety recall: fraction of genuinely disallowed content caught (per category; high recall on catastrophic categories).
- Over-refusal / false-positive rate: benign requests wrongly blocked — the precision side; measure it as seriously as misses.
- Adversarial/jailbreak benchmarks: red-team suites; attack success rate is the headline robustness metric. Anthropic's classifier evaluation — thousands of hours of incentivized human red-teaming against a live target (Sharma et al., 2025) — is the reference bar for "we tried hard to break it."
- LLM-as-judge + human audit on sampled decisions.
Think firstEvery generation now passes through multiple safety checks. How do you keep that from doubling latency?Commit to an answer, then revealHide
Make the checks small and concurrent: distilled per-category classifiers (not frontier models), input checks run in parallel with prompt processing, output checks streamed over tokens as they're generated rather than after the full response, and verdict caching for repeated content. Even a strong classifier stack costs real compute — Anthropic reported ~24% inference overhead, and safety that doubles latency gets quietly bypassed by product teams. Cheapness is part of the design requirement.
6. Operations & feedback
- Latency: run input/output classifiers in parallel with (or overlapped on) generation; keep them small; cache verdicts for identical content.
- Red-teaming loop: continuous adversarial testing feeds new jailbreaks into training — Constitutional Classifiers train on synthetic attack/benign data generated from a natural-language constitution, so a policy change regenerates the training set overnight.
- Policy versioning & audit: decisions are logged with reason codes; policy and thresholds are versioned; appeals overturn and relabel.
- Monitoring: track attack-success and over-refusal rates as live dashboards; new evasion patterns trigger targeted retraining.
Think firstProvenance marking: a C2PA-signed AI image gets screenshotted, an LLM's watermarked essay gets paraphrased. What survives — and what is watermarking actually for?Commit to an answer, then revealHide
Screenshots strip signed metadata; heavy paraphrase and re-encoding degrade statistical watermarks — no marking survives a determined adversary. But that's not the job. SynthID-Text watermarks by modifying only the sampling step, detectable without the model and cheap enough for production LLM traffic; C2PA gives cryptographic attribution on honest distribution paths. Together they make the default ecosystem legible (platform labeling, attribution, audit) and raise laundering costs — accountability infrastructure, not an unbreakable seal.
7. Tradeoffs & alternatives
| Choice | Alternative | Why |
|---|---|---|
| Layered defense-in-depth | Single moderation model | No one layer is robust to adversaries |
| Separate fast classifiers | Rely on model self-refusal only | Classifiers catch what alignment misses, and vice-versa |
| Calibrated per-category thresholds | One global threshold | Error costs differ hugely by category |
| Safeguard LLM (swappable taxonomy) | Fixed fine-tuned encoder | Policy changes without retraining; costs more per call |
| Watermark + provenance | No marking | Downstream accountability & deepfake mitigation |
8. Interviewer follow-ups
Users jailbreak the model with role-play prompts. How do you respond?
Diagnose the failure mode first (competing objectives vs. mismatched generalization — Wei et al.'s split), then layer: strengthen the system policy so user text can't override it, add adversarially-trained input/output classifiers for the attack family, and align the base model (RLHF/DPO) to refuse regardless of framing. Track attack-success rate on a red-team suite and feed new jailbreaks back into training — it's a continuous arms race, not a fixed rule set.
Safety is blocking legitimate requests (e.g. medical questions). Fix?
Over-refusal is a real failure. Calibrate per-category thresholds, use finer-grained policies (allow safe medical info, block dangerous instructions), and measure false-positive/over-refusal rate as a first-class metric — the benchmark to beat is strong jailbreak robustness at well under a 1% increase in benign refusals. A safety layer optimized only for recall will strangle the product; you're balancing recall against precision explicitly.
Why watermark outputs if classifiers already block bad content?
Different job: classifiers gate what's returned; watermarking/provenance makes returned content identifiable as AI-generated downstream (deepfake attribution, platform policy, regulation — Google deploys SynthID across text, image, and audio for exactly this). Blocking and provenance are complementary — one prevents, the other enables accountability for what is generated.
Key takeaways
- GenAI safety is a defense-in-depth pipeline — input moderation, hardened policy, output moderation, and model-level alignment — wrapped around every generation, compiled from a written policy taxonomy.
- It's adversarial and policy-driven: calibrate per-category thresholds, report robustness gained vs. benign refusals lost as a pair, and run a continuous red-teaming loop.
- Watermarking/provenance complements blocking — accountability infrastructure for the honest path, not an unbreakable seal.
References
- A Holistic Approach to Undesired Content Detection in the Real World: Markov et al., OpenAI, AAAI 2023; taxonomy design, labeling, and active learning behind a production moderation system.
- Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations: Inan et al., Meta, 2023; the safeguard-LLM pattern with prompt-swappable safety taxonomies.
- Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming: Anthropic, 2025; synthetic-data classifiers, the 3,000-hour red-team result, and the +0.38% refusal / ~24% overhead operating point.
- Jailbroken: How Does LLM Safety Training Fail?: Wei et al., NeurIPS 2023; the competing-objectives and mismatched-generalization failure modes of safety training.
- Scalable watermarking for identifying large language model outputs: Dathathri et al., Google DeepMind, Nature 2024 (SynthID-Text); production text watermarking via sampling modification.
- Content Credentials: C2PA Technical Specification: C2PA; the signed-manifest provenance standard for media content.
- DALL·E 3 System Card: OpenAI, 2023; a deployed multi-layer safety stack for an image-generation product.