Design a Multi-Agent Orchestration Pipeline
Track
Agentic System Design
Inspired by
AutoGen / CrewAI-style
Scale
planner + specialist workers · parallel · aggregation
Difficulty
Advanced
Some tasks are too big or too varied for one agent — generate a market report, migrate a codebase, run a content pipeline. A multi-agent system decomposes the work across specialized agents coordinated by an orchestrator — the architecture behind Anthropic's production Research system and frameworks like AutoGen and LangGraph. This is the interview about orchestration topology: how to decompose, delegate, run in parallel, and aggregate — without the complexity spiraling. Walk it through the agentic project loop.
1. Clarify the task (spine: task)
- Goal: complete a complex, multi-part task by coordinating specialized agents (e.g. researcher + writer + editor + fact-checker).
- When multi-agent is justified: distinct skills/contexts, parallelizable subtasks, or separation-of-duties (a checker reviewing a doer). Default to a single agent until one demonstrably can't cope.
- Success: the aggregate output is correct and coherent — coordination overhead paid for itself.
- Scope: the orchestration layer. Each worker reuses single-agent patterns.
Multi-agent is a cost, not a default
More agents = more latency, cost, and failure surface. Cognition's engineering position piece argues against multi-agent by default for exactly this reason: dispersed decision-making and fragmented context make fragile systems. Justify the topology by parallelism or genuinely distinct roles.
Think firstAnthropic measured its multi-agent Research system: it beat a single-agent Claude Opus 4 by 90.2% on their research eval — while burning ~15× the tokens of a chat. What do those two numbers jointly tell you about when this architecture pays?Commit to an answer, then revealHide
That multi-agent is an economics decision, not just an architecture decision. In Anthropic's analysis, token spend explains most of the performance variance — the system wins largely because parallel subagents deploy more total reasoning onto a parallelizable task. So it pays only when (a) the task value justifies ~15× chat cost, and (b) the task actually decomposes into parallel reads. A linear, low-value, or write-heavy task fails one of those conditions and belongs to a single agent.
2. Requirements & constraints
| Dimension | Target | Consequence |
|---|---|---|
| Decomposition | Clean subtask boundaries | Planner/orchestrator design |
| Parallelism | Independent subtasks concurrent | Fan-out/fan-in orchestration |
| Coherence | Aggregated output is consistent | Aggregation + conflict resolution |
| Cost/latency | Bounded despite many agents | Budgets per agent; avoid chatter loops |
Think firstYou've decided the task genuinely needs multiple agents. What are the candidate ways to wire them together — and what property of the task picks between them?Commit to an answer, then revealHide
Three basic topologies: orchestrator–worker (a planner delegates and aggregates — fits parallelizable, decomposable work; Anthropic's Research system and the orchestrator-workers pattern in Building Effective Agents), a sequential pipeline (stages hand off — fits inherently linear work), and generator–critic (iterate for quality — fits output where correctness is hard to get in one pass). The task's structure — parallel, linear, or quality-bound — picks the wiring; picking a topology for its novelty is how complexity spirals.
3. Orchestration topology (spine: orchestration)
The core design decision is the topology:
- Orchestrator–worker (recommended default): a lead agent decomposes the task, delegates subtasks to specialist workers, and aggregates results. Clear control flow, easy to trace. In Anthropic's production system the lead agent spawns 3–5 subagents in parallel, each searching with its own tools — cutting research time by up to 90% on complex queries.
- Sequential pipeline: stages hand off (research → draft → edit) — simple when work is linear.
- Debate / review: a generator and a critic iterate — good for quality-critical output.
The orchestrator: plans → fans out subtasks (parallel where independent) → each worker runs its own agent loop with a scoped context and tools → orchestrator aggregates, resolves conflicts, and verifies the combined result → replans if gaps remain. Delegation quality is the make-or-break skill: Anthropic found early orchestrators gave vague instructions ("research the semiconductor shortage") and got duplicated, drifting workers — each subtask needs an objective, an output format, and boundaries.
Think firstShould every worker agent see the full task context and every other worker's output?Commit to an answer, then revealHide
No — shared-everything multiplies cost, dilutes attention, and lets one agent's mistake contaminate the rest. Give each worker a scoped context (just its subtask and inputs), coordinate through a shared artifact store rather than chatty messages, and define structured interface contracts so the orchestrator aggregates typed results instead of parsing prose. (This is also Cognition's core objection to naive multi-agent: implicit decisions hide in context that other agents never see.)
4. Context isolation (spine: context)
- Scoped context per worker: each agent gets only what it needs — isolation keeps contexts small (latency/cost) and prevents cross-contamination.
- Shared artifacts: a common workspace/blackboard for results — workers write outputs to the store and the orchestrator reads them, rather than agents flooding each other with messages.
- Interface contracts: structured inputs/outputs between agents so aggregation is reliable, not free-text guessing.
Think firstResearchers annotated 1,600+ real multi-agent traces to build a failure taxonomy (MAST). The failures clustered in three places — none of them “the model wasn't smart enough.” Guess the three, and what they imply your evaluation must measure.Commit to an answer, then revealHide
The MAST taxonomy found 14 failure modes in three clusters: system design flaws (bad specs, role violations), inter-agent misalignment (workers ignoring each other, withholding or duplicating work), and task verification failures (no one checks the result). The implication: evaluating only the final output misses most of the causes. You need trajectory-level eval — was the decomposition sensible, did handoffs preserve intent, did verification actually run — with errors attributed to the responsible agent.
5. Evaluation (spine: evaluation)
- End-to-end outcome: is the final aggregate correct and coherent? (This is what matters — not any single agent's local success.)
- Per-agent + trajectory: which worker failed, was decomposition sensible, did agents loop or chatter unproductively? Most multi-agent failures are coordination failures, so this is where the signal lives.
- Attribution: trace errors to the responsible agent — essential for debugging a distributed system.
A generator–critic setup can improve quality, but beware groupthink and endless agreement — independent verification beats agents rubber-stamping each other.
Think firstWhat are the two signature failure modes unique to multi-agent systems — the ones a single agent can't have?Commit to an answer, then revealHide
Error propagation: one worker's hallucination flows through aggregation into the final output, laundered by the agents downstream, and runaway chatter — unbounded inter-agent messaging that burns budget without converging (MAST's inter-agent misalignment cluster, at 15× chat-level token prices). The countermeasures are structural: validate outputs at the aggregation boundary, and cap inter-agent turns and total spend.
6. Guardrails & safety (spine: guardrails)
- Error propagation: one worker's mistake can poison the aggregate — validate worker outputs before they're consumed downstream.
- Loop/cost control: cap inter-agent turns and total budget; multi-agent chatter is a classic runaway-cost failure.
- Consistent policy: all agents inherit the same safety guardrails; a worker can't be tricked into escalating privileges for the group.
Think firstA 30-minute, 20-agent run dies at minute 28 with one worker's API error. What must operations provide so this isn't a from-scratch restart — and how do you find which agent burned 80% of the budget?Commit to an answer, then revealHide
Durable execution and unified tracing. Anthropic's production lesson: long-running agents need checkpoint/resume so errors don't force expensive restarts from the beginning, and let the system handle retries around a saved state. For the budget question, a distributed trace with one span per agent (inputs, outputs, tool calls, tokens, wall time) makes per-agent spend a query instead of an investigation, and exposes the critical path that actually determines latency.
7. Operations (spine: operations)
- Distributed tracing: a unified trace across all agents (who did what, when, cost) — the only way to debug and to see the critical path.
- Cost/latency: parallelism cuts latency but multiplies cost — budget at the system level (recall the ~15× token multiplier) and prune agents that don't earn their spend.
- Checkpointing & durable execution: long multi-agent runs checkpoint so a single failure resumes rather than restarts; production systems also need rainbow-style deploys, since a prompt change mid-run strands in-flight agents.
8. Tradeoffs & alternatives
| Choice | Alternative | Why |
|---|---|---|
| Single agent + tools (first) | Multi-agent by default | Fewer moving parts; only add agents when parallelism/roles justify 15× cost |
| Orchestrator–worker | Free-for-all message passing | Clear control flow, traceable, parallelizable |
| Scoped per-agent context | Shared everything | Isolation controls cost and cross-contamination |
| Structured interfaces | Free-text handoffs | Reliable aggregation; fewer coordination errors |
9. Interviewer follow-ups
When is multi-agent actually worth it over one agent with tools?
When subtasks are genuinely parallel (fan-out speeds things up), require distinct skills/contexts that would bloat a single prompt, or benefit from separation-of- duties (an independent critic), and the task value covers the ~15× token cost. Anthropic's rule of thumb: multi-agent suits parallelizable read/research work; write-heavy work (most coding) coordinates poorly. If none apply, a single agent is simpler, cheaper, and easier to debug — start there.
One worker hallucinates and the final report inherits the error. Prevent it?
Validate worker outputs at the aggregation boundary (schema checks, an independent verifier/critic, cross-checking overlapping claims) rather than trusting them blindly. MAST classifies this as task-verification failure — one of the three dominant clusters, and the fix is structural: the orchestrator must be a gate, not just a concatenator.
Costs blew up — the agents kept messaging each other. Why?
Unbounded inter-agent communication. Cap turns and total budget, prefer a shared artifact/blackboard over chatty message passing, and use an orchestrator that drives control flow instead of agents free-forming. At 15× chat-level token usage, multi-agent chatter is the canonical runaway-cost mode.
Key takeaways
- Multi-agent is an economics + topology decision: orchestrator–worker (plan → fan-out → aggregate → verify) is the sane default, single-agent the saner starting point — the ~15× token multiplier must be earned by parallelizable, high-value work.
- Scoped context, structured interfaces, and output validation at aggregation prevent cost blowups and error propagation; most real failures are system design, misalignment, or verification problems (MAST), not model problems.
- Distributed tracing, checkpoint/resume, and per-agent + end-to-end evaluation are how you debug and trust a system of agents.
References
- How we built our multi-agent research system: Anthropic, orchestrator–worker in production: +90.2% over single-agent, ~15× token usage, delegation and durable-execution lessons.
- Why Do Multi-Agent LLM Systems Fail?: Cemri et al. (MAST), 14 failure modes across system design, inter-agent misalignment, and task verification, from 1,600+ annotated traces.
- AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation: Wu et al., the conversable-agents framework that popularized multi-agent orchestration.
- Don't Build Multi-Agents: Cognition, the counter-position: fragmented context and dispersed decisions make naive multi-agent fragile.
- How and when to build multi-agent systems: LangChain, reconciling the for/against positions into criteria for when the topology is justified.
- Building Effective AI Agents: Anthropic, the orchestrator-workers and evaluator-optimizer patterns this design builds on.