Design a Conversational AI Assistant at Scale
Agentic project loop
Task definition·Context & tools·Orchestration loop·Evaluation·Guardrails·Operations feedback

Design a Conversational AI Assistant at Scale

Track

Generative AI System Design

Inspired by

OpenAI / Anthropic

Scale

100M+ users · multi-turn · streaming · safety-critical

Difficulty

Intermediate

A general-purpose chat assistant (ChatGPT/Claude-style) is the flagship generative-AI product. The model is given — the system-design work is everything around it: managing conversation context, grounding with retrieval and memory, streaming responses, keeping it safe, and improving it from feedback. It reads through the agentic loop (context → orchestration → guardrails → operations).

1. Clarify the problem

  • Goal: a helpful multi-turn assistant answering open-ended questions and doing tasks, for a huge user base, safely and affordably.
  • Key sub-problems: long-conversation context management, factual grounding (reduce hallucination), safety, and cost/latency at scale.
  • Scope: the product system around the LLM (context, retrieval, memory, safety, serving). Base-model training and raw serving infra are adjacent (see the LLM inference-serving case).

Interview signal

The model is a given — signal comes from the context pipeline (what goes into the prompt), safety, and the feedback loop. Treat it as a product/systems problem, not a modeling one.

Think firstA raw pretrained LLM will happily 'chat' if you prompt it with a dialogue transcript — GPT-3 could in 2020. Name three concrete failures you'd see in week one of shipping that as a product.Commit to an answer, then reveal

(1) It completes text instead of following instructions — verbose, off-task, continuing the pattern rather than helping; this is exactly the gap RLHF closed, where a preference-tuned 1.3B model was preferred by humans over the raw 175B GPT-3. (2) It asserts confident falsehoods with no grounding in anything checkable. (3) It produces harmful content on request — nothing in a pretraining objective says no. The whole product is the machinery that fixes these three: alignment, grounding, guardrails.

2. Requirements & constraints

DimensionTargetConsequence
LatencyStream first token fastStreaming; model tiering
ContextLong multi-turn conversationsSummarization/truncation; memory retrieval
GroundingReduce hallucination, citeOptional RAG + tools; groundedness metric
SafetyBlock harmful content/misuseInput/output moderation, jailbreak defense
CostFrontier models are expensiveRoute easy turns to smaller models; cache

These aren't ad-hoc: Google's LaMDA formalized dialog quality as sensibleness, specificity, interestingness plus safety and groundedness: the fraction of external-world claims supported by authoritative sources, and showed fine-tuning improves all of them beyond what scale alone delivers (Thoppilan et al., 2022).

Think firstA conversation reaches turn 200; the context window fits 20. What's your strategy — and why is plain truncation wrong?Commit to an answer, then reveal

Truncation silently drops the thread — the user's name, the constraint from turn 3, the decision from turn 50. The working pattern is layered: keep recent turns verbatim, roll older turns into a running summary, and move durable facts into retrievable memory pulled in only when relevant. The context window is a budget to allocate, not a buffer to overflow.

3. Context & memory (spine: context)

The heart of the product is what fills the prompt:

  • Conversation window: recent turns verbatim; older turns summarized to fit the context budget without losing the thread.
  • Long-term memory: durable user facts/preferences persisted across conversations and retrieved into context when relevant. This is how ChatGPT's memory works in production: explicit "saved memories" plus reference to past chat history, with user-visible controls to inspect, correct, or disable it — memory that users can't audit destroys trust (OpenAI, 2024).
  • Grounding tools: retrieval (web/docs) and tools (calculator, code) when a turn needs facts or actions — with citations. LaMDA's groundedness fine-tuning worked the same way: teach the model to call a toolset (search, calculator) and revise its answer against the results.
  • System prompt: the assistant's persona, policies, and capabilities.
Think firstYou have a 100k-token window, so you stuff every retrieved memory and the full history in 'to be safe.' Retrieval is perfect. Why do answers still miss facts that are sitting right there in the prompt?Commit to an answer, then reveal

Because attention over long contexts is not uniform: models reliably use information at the beginning and end of the context and degrade sharply when the relevant fact sits in the middle — the "lost in the middle" U-curve, which holds even for explicitly long-context models (Liu et al., 2023). A long window is capacity, not comprehension. So the orchestrator must curate and position: retrieve less, rank it, and place the critical material near the edges of the prompt.

4. Orchestration (spine: orchestration)

Per turn: assemble context (system prompt + retrieved memory + summarized history + grounding), positioning the most relevant material where the model actually attends → decide if tools/retrieval are needed → generate a streamed response → post-process (safety, citations). Decoding strategy (temperature, top-p) trades determinism vs. creativity; tool-calling turns loop through act→observe like an agent.

Decoding: Temperature & Nucleus (top-p) Sampling

0.000.100.200.310.41mat41%floor22%sofa14%roof5%table9%bed6%grass2%moon1%
tokens kept8 / 8
nucleus mass100%
entropy (bits)2.36

Next-token distribution after “The cat sat on the ___”. Temperature reshapes the distribution (low → peaky/greedy, high → flat/creative); top-p keeps only the smallest set of tokens whose probability sums to p (greyed-out bars are excluded), then renormalises the survivors.

Sampling parameters shape the assistant's "voice" — low temperature for factual tasks, higher for brainstorming; the product often adapts them per intent.

Think firstOutputs are open-ended prose — there's no accuracy number. How do you decide a new model version is actually better?Commit to an answer, then reveal

Preference win-rate: sample real prompts, generate with old and new, and have humans (and a calibrated LLM judge) pick blind winners — the same methodology InstructGPT used to show its 1.3B model beating 175B GPT-3. Add targeted checks the win-rate hides — groundedness when retrieval is used (LaMDA's metric), refusal correctness in both directions, and watch live signals (thumbs, regeneration rate) after rollout. No single number; a panel of them.

5. Evaluation (spine: evaluation)

  • Quality: helpfulness/correctness via human preference and LLM-as-a-judge on curated + live prompts; track win-rate against the previous version (Ouyang et al., 2022).
  • Faithfulness: groundedness when retrieval is used — fraction of factual claims supported by the cited sources.
  • Safety: refusal correctness in both directions — refuses harmful, doesn't over-refuse benign — plus jailbreak resistance under red-teaming.
  • Product metrics: retention, thumbs up/down, regeneration rate.
Deep dive
LLM-as-a-judge for open-ended quality
Includes runnable notebook
Deep dive
Generation-quality metrics
Includes runnable notebook
Think firstYour input filter blocks 99% of harmful requests, yet red-teamers extract harmful output within an hour. What did the single-filter design get wrong?Commit to an answer, then reveal

It assumed harm arrives labeled. Jailbreaks smuggle intent past input classifiers (role-play, encoding, multi-turn setup), and harm can emerge in the output even from a benign-looking prompt. Defense has to be layered — input classification, alignment training in the model itself, output classification, and injected-content hygiene, because each layer catches a different failure shape. This is why production stacks run a dedicated safeguard model on both the prompt and the response, not a keyword filter on the way in.

6. Safety & guardrails (spine: guardrails)

  • Input & output moderation: a dedicated classifier on both sides of generation. Llama Guard is the canonical open pattern — an LLM fine-tuned against a safety taxonomy that classifies prompts and responses, matching or beating purpose-built moderation APIs, with the taxonomy adaptable per product (Inan et al., 2023).
  • Alignment in the model: Constitutional AI trains harmlessness from AI-generated critiques and revisions against explicit principles (RLAIF), yielding a non-evasive assistant that explains refusals and engages with hard questions instead of stonewalling (Bai et al., 2022).
  • Jailbreak/prompt-injection defense: user text (and any retrieved content) is untrusted and must not override the system policy.
  • Refusal calibration: over-refusal is a product failure too — measure both directions, not just blocked-harm rate.
Deep dive
Jailbreak & injection defenses
Includes runnable notebook
Think firstFrontier-model inference on every turn for 100M users is ruinous. Where does most of the cost saving come from?Commit to an answer, then reveal

From admitting most turns are easy: route them to a small, cheap model and escalate only the hard ones — plus prefix caching for the shared system prompt and response caching for repeated queries. Model tiering typically saves more than any inference optimization, because the cheapest token is one a frontier model never generates. The InstructGPT result is the reason tiering works at all: a well-tuned small model is genuinely competitive on the easy mass of traffic.

7. Operations (spine: operations)

  • Serving: streaming, model tiering (small model for easy turns, frontier for hard), prompt/prefix caching — heavy overlap with the LLM inference-serving case.
  • Observability: trace prompts, retrievals, tool calls, tokens, cost, latency, and feedback per conversation.
  • Feedback loop: thumbs down, regenerations, and preference data drive RLHF/DPO alignment and prompt/policy updates — the same human-preference machinery that built the assistant keeps improving it.

8. Tradeoffs & alternatives

ChoiceAlternativeWhy
Summarize + retrieve memoryFull history in a huge windowLong-context attention degrades mid-prompt; curation beats capacity
Model tieringOne frontier model for allMost turns are easy; a tuned small model handles them well
Optional RAG + toolsRely on parametric knowledgeGroundedness is measurable; parametric memory is stale
Safeguard model on input and outputInput keyword filterJailbreaks bypass input-only checks; harm emerges in outputs
Non-evasive refusals (CAI-style)Blanket refusalOver-refusal is a failure mode; users need engagement, not evasion

9. Interviewer follow-ups

A conversation exceeds the context window. What happens?

Summarize older turns into a running synopsis, keep recent turns verbatim, and retrieve any specifically-referenced earlier content. Naive truncation drops the thread; a rolling summary preserves it within budget. Durable facts live in cross-conversation memory (ChatGPT's saved-memories pattern) and are retrieved when relevant — with user-facing controls to view, edit, and disable, because unauditable memory is a trust and privacy liability.

How do you cut cost when most conversations are trivial?

Route by difficulty: a small/cheap model handles easy turns and escalates hard ones to a frontier model, plus prompt/prefix caching for the shared system prompt. Most traffic is easy, so tiering is the biggest cost lever — measure the per-tier quality delta with win-rate evals to set the routing threshold, and re-measure it every time either model changes.

A user tries to jailbreak the assistant into harmful output. Defenses?

Layered: a safeguard classifier on the input, a system policy user text can't override, alignment training in the model (RLHF/Constitutional AI for refusal behavior), and a classifier on the output — the Llama-Guard-style both-sides pattern. Treat user and retrieved text as untrusted; no single layer holds, so defense-in-depth plus continuous red-teaming, with refusal quality measured in both directions so the fix doesn't create an over-refusing assistant.

Key takeaways

  • A chat assistant is a product/systems problem around a given model — the context pipeline, safety, and feedback loop are where it's won; alignment (RLHF) is what turned raw LLMs into products at all.
  • Manage long conversations with summarization + retrieved, user-auditable memory, and curate context rather than stuffing it — long windows suffer measurable mid-prompt blindness.
  • Run safeguard classification on both input and output, train non-evasive refusals, and make scale affordable with model tiering + caching — a well-tuned small model handles the easy mass of traffic.

References