Design an AI Writing Assistant
Track
Generative AI System Design
Inspired by
Jasper / Notion AI
Scale
long-form · brand-voice control · grounded · preference-tuned
Difficulty
Intermediate
A writing assistant (marketing copy, blog posts, emails, docs) is a controllable long-form text generation product. The model is given — the system-design work is control (tone, style, brand voice, format), grounding (facts, no hallucination), long-form coherence, and tuning quality from human preference. It reads through the ML loop's objective → evaluation → feedback stages.
1. Clarify the problem
- Goal: help users produce high-quality long-form text in a controlled style/format, grounded in provided facts/sources, efficiently.
- Control axes: tone, brand voice, length, format/structure, audience — the product's differentiator over raw chat.
- Grounding: for factual content, cite sources / stay faithful to provided material — a hallucinated statistic in a customer's post is their reputation.
- Scope: the generation product (control + grounding + long-form + feedback).
Interview signal
The differentiator isn't the model — it's controllability (reliable tone/ brand/format) and grounding. Frame it as steering and constraining a strong base model, and tuning it with preference data.
Think firstAny frontier model already writes fluent prose. So where does a writing product actually add value — what are you being paid for?Commit to an answer, then revealHide
Everything fluency doesn't cover: reliable control (this brand's voice, this format, every time — not "usually"), grounding (claims tied to the user's sources), and long-form structure (a 2,000-word piece that doesn't drift). The base model is a commodity; the control system is the product — which is why the requirements table below is about control, grounding, coherence, and preference, and never mentions raw fluency.
2. Requirements & constraints
| Dimension | Target | Consequence |
|---|---|---|
| Control | Reliable tone/brand/format | Structured prompting, style conditioning, fine-tuning |
| Grounding | Faithful to sources; cite | RAG over user material; faithfulness checks |
| Long-form | Coherent multi-section docs | Outline-then-expand; section planning |
| Quality | Matches human preference | Preference tuning (RLHF/DPO) on product data |
| Latency | Inline suggestions feel instant | Small specialized models for the edit path |
Think firstGrammarly fires a suggestion on nearly every sentence you type; Jasper generates a 2,000-word draft on demand. Same product category — but what's architecturally different, and can one model serve both?Commit to an answer, then revealHide
They sit at opposite ends of an edit ↔ generate spectrum with opposite constraints. Inline correction needs tens-of-ms latency and surgical, local edits — Grammarly's GECToR does it as sequence tagging (predict per-token edit operations), not generation, running up to 10× faster than a seq2seq rewriter. Draft generation tolerates seconds and needs global planning — a big autoregressive LLM. One model serves both badly; real products tier: small specialized editors on the hot path, a large model for drafting.
3. GenAI problem framing
Controllable, grounded long-form generation on top of a base LLM — but "writing assistance" is a family of tasks along the edit↔generate spectrum:
- Correction/editing: local, latency-critical; Grammarly frames it as tagging, not rewriting — a BERT-class encoder predicts token-level edit operations, up to 10× faster inference than seq2seq GEC (Omelianchuk et al., 2020).
- Instructed revision: "make this simpler," "make it more formal" — Grammarly's CoEdIT fine-tunes on 82K task-specific edit instructions and matches much larger general LLMs while being ~60× smaller (Raheja et al., 2023) — task-specific tuning beats scale for a fixed task menu.
- Drafting: open-ended long-form generation, where control comes from (a) structured prompting (system prompt + style exemplars), (b) optional fine-tuning on a brand's corpus, and (c) decoding choices; grounding from RAG over the user's sources.
Pre-generative baseline? Templates + human writers — inflexible or slow. Generative writing adapts to any prompt while (ideally) staying on-brand.
Think firstAsk a strong model for a 2,000-word article in a single generation. What specifically degrades as it writes — and what structure fixes it?Commit to an answer, then revealHide
Three failure modes compound: drift (by paragraph twelve the piece has wandered off the brief, because each token conditions on its own possibly- drifting output), repetition (the same argument resurfaces re-phrased), and structural imbalance (a bloated intro, a rushed ending — the model can't budget words it hasn't generated). The fix is plan-then-write: generate an outline first, expand sections conditioned on the outline and prior sections, then a revision pass. The outline is also the product's steering wheel — users edit it before committing to prose.
4. Model & generation (spine: objective)
- Base + light adaptation: a strong instruction-tuned base; a LoRA adapter per brand where prompting isn't reliable enough — low-rank updates train ~10,000× fewer parameters than full fine-tuning with matching quality, so per-customer voice adapters are economically feasible (Hu et al., 2021).
- Decoding: moderate temperature/top-p for fluent-but-controlled prose; lower for factual/brand-safe copy, higher for ideation — often exposed to the user as "creativity."
- Structure: outline → section drafts → revise, so long-form stays coherent and the user can steer at the outline level.
Decoding: Temperature & Nucleus (top-p) Sampling
Next-token distribution after “The cat sat on the ___”. Temperature reshapes the distribution (low → peaky/greedy, high → flat/creative); top-p keeps only the smallest set of tokens whose probability sums to p (greyed-out bars are excluded), then renormalises the survivors.
Think firstA customer uploads brand guidelines and source documents. Through what mechanisms do those actually shape the generated text?Commit to an answer, then revealHide
Three, in increasing strength: retrieval pulls the sources and guidelines into context with a requirement to cite; style conditioning puts voice exemplars and explicit constraints (tone, reading level, format) in the prompt; and where prompting isn't reliable enough, a LoRA adapter fine-tuned on the brand's corpus bakes the voice in. Format guarantees come from constrained decoding, not hope.
5. Grounding & control
- RAG: retrieve the user's provided sources/brand guidelines into context; generate with citations; verify claims are supported. Retrieval-augmented generation produces measurably more specific and factual text than relying on parametric memory alone (Lewis et al., 2020).
- Style conditioning: brand-voice exemplars and explicit constraints (tone, reading level, format) in the prompt or a fine-tuned adapter.
- Structured output: enforce format (sections, bullet limits, JSON for structured docs) via constrained decoding / schemas.
Think firstFluency is table stakes — every draft reads well. What do you actually measure to know the product is working?Commit to an answer, then revealHide
The things fluency hides: control adherence (did it match the requested tone/length/format — a checkable constraint), faithfulness (is every claim supported by the provided sources), and the bluntest product signal of all — edit rate: how much of the draft users rewrite before shipping it. A beautiful draft that gets 80% rewritten failed.
6. Evaluation (spine: evaluation)
- Human preference / LLM-as-judge: the primary signal for quality, tone adherence, and helpfulness; track win-rate vs. the previous version.
- Faithfulness: when grounded, are claims supported by the sources? (judge + human audit).
- Control adherence: does output match requested tone/length/format? — a measurable constraint check, not a vibe.
- Product metrics: acceptance/edit rate — how much users change the draft before using it.
Think firstYou have 100k sessions where users accepted, rejected, or edited drafts. How do you turn that exhaust into a better model — and what bias lurks in it?Commit to an answer, then revealHide
Edits are implicit preference pairs: the draft the model produced vs. the version the user shipped — exactly the (rejected, chosen) format preference tuning consumes. DPO makes this practical for a product team: it optimizes the preference objective directly with a classification-style loss, no reward model or RL loop to operate. The lurking bias: you only observe edits from users who didn't churn, and heavy editors aren't average users, so weight and audit the data, or the model converges to the taste of your most patient customers.
7. Feedback & operations (spine: feedback)
- Preference loop: edits, regenerations, and thumbs become preference pairs. The InstructGPT result is the existence proof for how far this goes: a 1.3B model tuned on human preferences was preferred over the raw 175B GPT-3 (Ouyang et al., 2022) — preference data beats two orders of magnitude of scale for "does what the user meant." DPO delivers most of this with a simple classification loss instead of a full RLHF pipeline (Rafailov et al., 2023).
- Serving: streaming for responsiveness; model tiering (fast editor models on the inline path, a large model for drafts); caching of retrieved brand material.
- Guardrails: plagiarism/verbatim-source checks, disallowed-content filters, and disclosure that content is AI-assisted.
8. Tradeoffs & alternatives
| Choice | Alternative | Why |
|---|---|---|
| Plan-then-write | One long generation | Long-form coherence; outline as user steering surface |
| Prompt + LoRA for voice | Full fine-tune | ~10,000× fewer trained params; adapter-per-brand is feasible |
| Task-specific editor models | One frontier model for all | 60× smaller matches larger models on a fixed task menu; latency |
| RAG grounding | Rely on parametric knowledge | Faithfulness, freshness, citeability |
| DPO on product data | Full RLHF pipeline | Same preference signal, far simpler to train and operate |
9. Interviewer follow-ups
Output is fluent but off-brand. How do you enforce voice?
Layer control: brand-voice exemplars and explicit style constraints in the prompt first; if that's insufficient, a LoRA adapter fine-tuned on the brand's corpus for a consistent voice. Measure tone/voice adherence explicitly (rubric-based LLM-judge + human audit) so "off-brand" is caught pre-ship, not by users. The escalation ladder is prompt → exemplars → adapter — each step costs more and drifts less.
The assistant invents statistics in a factual article. Fix?
Ground it: retrieve the user's sources and require claims to be supported and cited, run a faithfulness check (judge over claim vs. source), and for unsupported claims, hedge or omit rather than fabricate. Ungrounded long-form is a hallucination magnet — RAG plus a faithfulness gate is the standard remedy, and it's also what makes output citeable, which users need for trust.
How do you keep a 2000-word piece coherent and on-topic?
Plan-then-write: generate an outline, expand each section conditioned on the outline + prior sections, then a revision pass for consistency. One monolithic generation drifts, repeats, and misbudgets structure; explicit planning keeps long-form on track and lets the user steer at the outline level before any prose exists.
Key takeaways
- A writing assistant spans an edit ↔ generate spectrum: sequence-tagging editors (10× faster than seq2seq) on the latency-critical path, large models for drafting — the value is control, faithfulness, and structure, not the base model.
- Use plan-then-write for coherence, RAG for grounding, and prompt + LoRA adapters for brand voice at per-customer cost.
- Turn user edits into preference pairs (DPO makes this operationally cheap; InstructGPT proved preference data beats raw scale) and measure control adherence, faithfulness, and edit rate, not fluency.
References
- GECToR — Grammatical Error Correction: Tag, Not Rewrite: Omelianchuk et al. (Grammarly), BEA 2020; sequence-tagging correction with up to 10× faster inference than seq2seq.
- Grammatical Error Correction: Tag, Not Rewrite: Grammarly Engineering Blog; the production story behind GECToR's design.
- CoEdIT: Text Editing by Task-Specific Instruction Tuning: Raheja et al. (Grammarly), EMNLP Findings 2023; 82K edit instructions, competitive at ~60× smaller scale.
- Training language models to follow instructions with human feedback: Ouyang et al. (OpenAI), NeurIPS 2022; InstructGPT — preference-tuned 1.3B preferred over raw 175B GPT-3.
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model: Rafailov et al., NeurIPS 2023; preference tuning as a classification loss, no RL loop.
- LoRA: Low-Rank Adaptation of Large Language Models: Hu et al. (Microsoft), 2021; ~10,000× fewer trainable parameters, the basis of per-brand voice adapters.
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks: Lewis et al., NeurIPS 2020; retrieval yields more specific, factual generation than parametric memory.