Design a Real-Time Voice Agent
Track
Agentic System Design
Inspired by
OpenAI Realtime / Retell
Scale
<800ms turn latency · streaming ASR/TTS · barge-in
Difficulty
Advanced
A voice agent (think an AI phone receptionist or hands-free assistant) is the agentic interview where latency is the architecture. Human conversation turns over in a few hundred milliseconds, so the entire listen → think → speak loop must stream and overlap. Everything else (tools, guardrails) is standard; the differentiator is the real-time pipeline. Walk it through the agentic project loop.
1. Clarify the task (spine: task)
- Goal: hold a natural spoken conversation that accomplishes a goal (book an appointment, answer a question, take an order), possibly calling tools.
- The bar is responsiveness: sub-second turn latency and natural turn-taking (including barge-in — the user interrupting).
- Modality bridge: audio in → text reasoning → audio out, each stage streamed.
- Scope: the real-time voice pipeline + agent loop. Reuses tool-use/guardrail patterns from the support agent.
Interview signal
"Latency is the constraint that shapes everything — I'll stream and overlap ASR, LLM, and TTS rather than run them in sequence." Leading with the latency budget is the signal that you've built voice, not just chat.
Think firstPut a number on it: how long can the agent pause after the caller stops talking before the conversation feels broken — and what does that number force architecturally?Commit to an answer, then revealHide
Human turn-taking gaps run on the order of ~200ms, and production platforms converge on roughly the same budget: Vapi's published targets are p50 under 500ms and p95 under 800ms; past ~800ms callers assume the line dropped and start talking over the agent. No LLM round-trip plus ASR plus TTS fits that budget run sequentially, so the number itself dictates a streaming, overlapped pipeline before you've chosen a single model.
2. Requirements & constraints
| Dimension | Target | Consequence |
|---|---|---|
| Turn latency | ~500 ms p50, < 800 ms p95 | Stream + overlap ASR/LLM/TTS; speak first tokens early |
| Turn-taking | Detect end-of-speech, allow barge-in | VAD, endpointing, interruptible TTS |
| Robustness | Noise, accents, disfluency | Strong streaming ASR; tolerate partial transcripts |
| Scale | Many concurrent calls | Efficient serving; telephony integration |
Think firstAdd up a sequential pipeline: full ASR (~300ms after end of speech) → LLM response (~1–2s) → TTS synthesis (~300ms). Does it fit the 800ms budget — and if not, where does the time come back from?Commit to an answer, then revealHide
It doesn't — sequential is 2–3× over budget before network. The time comes back from overlap: ASR streams partials while the user is still talking, the LLM starts on the near-final transcript before the turn formally ends, and TTS speaks the first sentence while the rest is still generating. No stage waits for the previous one to finish — with all three streaming, sub-second end-to-end is achievable with today's components (sub-300ms streaming ASR, ~75ms TTS inference).
3. Context & tools (spine: context)
- Pipeline components: streaming ASR (speech→text), the LLM (reasoning
- tool calls), streaming TTS (text→speech), and a VAD/endpointer for turn boundaries. Whisper-class models set the robustness bar for ASR — trained on 680k hours of weakly supervised audio precisely to survive accents, noise, and jargon.
- Tools: the same typed function calls as a text agent (lookup, book, etc.), invoked mid-conversation.
- Context: conversation history + task state; kept compact for latency.
Think firstOpenAI's Realtime API processes audio in and out through one speech-to-speech model — no transcripts in the loop. Why would you still choose the three-stage cascade for a production phone agent?Commit to an answer, then revealHide
Speech-to-speech wins on latency and naturalness — one model, no pipeline hops, prosody preserved. The cascade wins on control: you can swap the best ASR/LLM/TTS per language and price point, inspect an exact transcript at every stage (audit, compliance, debugging), and apply text-based guardrails before anything is spoken. For regulated, tool-heavy phone work the inspectable text seam usually pays for its milliseconds; for expressive consumer voice, speech-to-speech increasingly wins.
4. Orchestration (spine: orchestration)
The loop is listen → understand → act/speak, but streamed and overlapped:
- ASR emits partial transcripts as the user speaks; the endpointer decides when the turn ends — tune it too eager and you interrupt callers, too lazy and every turn feels sluggish. Endpointing is the most underrated latency component.
- The LLM begins reasoning on the (near-)final transcript and streams tokens; TTS starts speaking the first sentence before the full response is generated.
- Barge-in: if the user starts talking, immediately stop TTS, flush the queued audio, and re-listen — interruption handling is core, not an edge case.
- Tool calls happen mid-turn; fill dead air with a brief "let me check that."
Think firstWhich single number best predicts whether the agent feels responsive — total response time, or something else?Commit to an answer, then revealHide
Time-to-first-audio. Users perceive the pause before the agent starts speaking, not how long the full answer takes — a response that begins in 400ms and talks for ten seconds feels snappy; one that starts after two seconds feels broken regardless of length. This is why every serious stack streams TTS by sentence and why vendors publish time-to-first-audio, not completion time. Optimize and monitor the start, not the total.
5. Evaluation (spine: evaluation)
- Latency: p50/p95 turn latency and time-to-first-audio — the primary metric, broken down per stage (endpointing, ASR final, LLM first token, TTS first byte).
- ASR quality: word error rate on your domain audio (accents, noise, jargon) — benchmark WER and production WER routinely diverge.
- Task success: did the call accomplish its goal? (booking completed, question answered) — outcome eval on transcripts.
- Conversation quality: interruption handling, naturalness, no talking over the user — often human-rated.
Think firstASR might mis-hear a payment amount or an address. Confirm every detail with the user, then?Commit to an answer, then revealHide
No — every confirmation costs a full slow turn and makes the agent feel robotic. The policy is selective: confirm only high-stakes, hard-to-reverse slots (payments, identity, addresses), triggered by ASR confidence, and let low-stakes details flow. Safety spent uniformly is latency wasted; spend it where an error actually hurts.
6. Guardrails & safety (spine: guardrails)
- Same action guardrails as any tool-using agent: permissioned tools, HITL for risky actions, injection defense on transcribed content.
- Voice-specific: graceful failure on ASR errors (confirm critical details: "did you say 3pm?"), and clear disclosure that it's an AI.
- Latency vs. safety: confirmations add turns — reserve them for high-stakes slots (payment, identity), not everything.
Think firstTuesday's deploy doubled p95 turn latency. The pipeline has four stages plus network. What must already be in place for this to be a ten-minute investigation instead of a week?Commit to an answer, then revealHide
Per-stage latency tracing on every turn: endpointing delay, ASR finalization, LLM time-to-first-token, TTS time-to-first-byte, and network, logged as a span per turn. Aggregate p95 says "slower"; only the stage breakdown says where — a regressed endpointer looks identical to a slow LLM in the end-to-end number. Voice is a distributed system with a 800ms SLO; you operate it like one.
7. Operations (spine: operations)
- Serving: low-latency, high-concurrency inference — continuous batching, optimized ASR/TTS, and often speculative decoding to cut LLM latency.
- Tracing: log audio, transcripts, and per-stage latencies per turn, plus tool calls — the per-stage breakdown is how you find the bottleneck.
- Telephony: integrate with SIP/PSTN; handle jitter, packet loss, echo — and co-locate ASR/LLM/TTS regions, because a pipeline that hops clouds donates its budget to the network.
8. Tradeoffs & alternatives
| Choice | Alternative | Why |
|---|---|---|
| Streamed, overlapped pipeline | Sequential ASR→LLM→TTS | Sequential blows the latency budget |
| Cascaded ASR+LLM+TTS | End-to-end speech-to-speech (Realtime API) | Cascade is debuggable/controllable; s2s is lower-latency & more natural |
| Speak first tokens early | Wait for full response | Time-to-first-audio dominates perceived latency |
| Confirm only high-stakes slots | Confirm everything | Every confirmation adds a slow turn |
9. Interviewer follow-ups
The user interrupts mid-sentence. What happens?
Barge-in handling: VAD detects incoming speech, TTS stops immediately and flushes the queued audio, and the agent re-listens, incorporating the interruption. The context must also be truncated to what was actually spoken — the user is responding to the words they heard, not the paragraph you generated. Talking over the user is the most unnatural failure; interruption is a first-class path.
Where does the 800ms go, and how do you cut it?
Endpointing (waiting to be sure the turn ended), ASR finalization, LLM time-to-first-token, TTS time-to-first-audio, plus network per hop. Cut it by streaming partial transcripts, starting LLM reasoning before end-of-speech is certain, speaking the first sentence early, using fast TTS (~75ms-class models), and co-locating stages. Trace per-stage latency to find the real bottleneck — it's usually endpointing or the LLM, not TTS.
ASR mis-hears a critical detail (an address, a date). How do you handle it?
Confirm high-stakes slots explicitly ('I heard 742 Evergreen — correct?') using ASR confidence to decide when to confirm, and design tools to tolerate correction. Blindly acting on a low-confidence transcript for an irreversible action is the risk; targeted confirmation buys safety without slowing every turn.
Key takeaways
- For a voice agent, latency is the architecture — target ~500ms p50 / 800ms p95 per turn, stream and overlap ASR, LLM, and TTS; time-to-first-audio is the metric that matters.
- Barge-in, endpointing, and turn-taking are core interaction paths, not edge cases, and endpointing is the most underrated latency component.
- Choose cascade for control (inspectable transcripts, per-stage swaps) or speech-to-speech for latency/naturalness; either way, add ASR-confidence confirmations for high-stakes slots and serve with continuous batching.
References
- Introducing the Realtime API: OpenAI, single-model speech-to-speech vs. chained pipelines and why it cuts latency and preserves prosody.
- Introducing gpt-realtime and Realtime API updates for production voice agents: OpenAI, the production speech-to-speech alternative to the cascade.
- Robust Speech Recognition via Large-Scale Weak Supervision: Radford et al. (Whisper), the 680k-hour weak-supervision recipe behind robust ASR on accents and noise.
- Speech Latency Solutions: Complete Guide to Sub-500ms Voice AI: Vapi, published p50 under 500ms / p95 under 800ms turn-latency targets and the per-stage budget.
- Voice Agent Architecture: STT, LLM, and TTS Pipelines Explained: LiveKit, streaming/overlap pipeline design and component latency breakdown.
- Meet Flash: ElevenLabs, ~75ms TTS model inference for conversational agents.