Design a Real-Time Voice Agent
Agentic project loop
Task definition·Context & tools·Orchestration loop·Evaluation·Guardrails·Operations feedback

Design a Real-Time Voice Agent

Track

Agentic System Design

Inspired by

OpenAI Realtime / Retell

Scale

<800ms turn latency · streaming ASR/TTS · barge-in

Difficulty

Advanced

A voice agent (think an AI phone receptionist or hands-free assistant) is the agentic interview where latency is the architecture. Human conversation turns over in a few hundred milliseconds, so the entire listen → think → speak loop must stream and overlap. Everything else (tools, guardrails) is standard; the differentiator is the real-time pipeline. Walk it through the agentic project loop.

1. Clarify the task (spine: task)

  • Goal: hold a natural spoken conversation that accomplishes a goal (book an appointment, answer a question, take an order), possibly calling tools.
  • The bar is responsiveness: sub-second turn latency and natural turn-taking (including barge-in — the user interrupting).
  • Modality bridge: audio in → text reasoning → audio out, each stage streamed.
  • Scope: the real-time voice pipeline + agent loop. Reuses tool-use/guardrail patterns from the support agent.

Interview signal

"Latency is the constraint that shapes everything — I'll stream and overlap ASR, LLM, and TTS rather than run them in sequence." Leading with the latency budget is the signal that you've built voice, not just chat.

Think firstPut a number on it: how long can the agent pause after the caller stops talking before the conversation feels broken — and what does that number force architecturally?Commit to an answer, then reveal

Human turn-taking gaps run on the order of ~200ms, and production platforms converge on roughly the same budget: Vapi's published targets are p50 under 500ms and p95 under 800ms; past ~800ms callers assume the line dropped and start talking over the agent. No LLM round-trip plus ASR plus TTS fits that budget run sequentially, so the number itself dictates a streaming, overlapped pipeline before you've chosen a single model.

2. Requirements & constraints

DimensionTargetConsequence
Turn latency~500 ms p50, < 800 ms p95Stream + overlap ASR/LLM/TTS; speak first tokens early
Turn-takingDetect end-of-speech, allow barge-inVAD, endpointing, interruptible TTS
RobustnessNoise, accents, disfluencyStrong streaming ASR; tolerate partial transcripts
ScaleMany concurrent callsEfficient serving; telephony integration
Think firstAdd up a sequential pipeline: full ASR (~300ms after end of speech) → LLM response (~1–2s) → TTS synthesis (~300ms). Does it fit the 800ms budget — and if not, where does the time come back from?Commit to an answer, then reveal

It doesn't — sequential is 2–3× over budget before network. The time comes back from overlap: ASR streams partials while the user is still talking, the LLM starts on the near-final transcript before the turn formally ends, and TTS speaks the first sentence while the rest is still generating. No stage waits for the previous one to finish — with all three streaming, sub-second end-to-end is achievable with today's components (sub-300ms streaming ASR, ~75ms TTS inference).

3. Context & tools (spine: context)

  • Pipeline components: streaming ASR (speech→text), the LLM (reasoning
    • tool calls), streaming TTS (text→speech), and a VAD/endpointer for turn boundaries. Whisper-class models set the robustness bar for ASR — trained on 680k hours of weakly supervised audio precisely to survive accents, noise, and jargon.
  • Tools: the same typed function calls as a text agent (lookup, book, etc.), invoked mid-conversation.
  • Context: conversation history + task state; kept compact for latency.
Deep dive
Continuous batching — high-throughput LLM serving
Includes runnable notebook
Think firstOpenAI's Realtime API processes audio in and out through one speech-to-speech model — no transcripts in the loop. Why would you still choose the three-stage cascade for a production phone agent?Commit to an answer, then reveal

Speech-to-speech wins on latency and naturalness — one model, no pipeline hops, prosody preserved. The cascade wins on control: you can swap the best ASR/LLM/TTS per language and price point, inspect an exact transcript at every stage (audit, compliance, debugging), and apply text-based guardrails before anything is spoken. For regulated, tool-heavy phone work the inspectable text seam usually pays for its milliseconds; for expressive consumer voice, speech-to-speech increasingly wins.

4. Orchestration (spine: orchestration)

The loop is listen → understand → act/speak, but streamed and overlapped:

  • ASR emits partial transcripts as the user speaks; the endpointer decides when the turn ends — tune it too eager and you interrupt callers, too lazy and every turn feels sluggish. Endpointing is the most underrated latency component.
  • The LLM begins reasoning on the (near-)final transcript and streams tokens; TTS starts speaking the first sentence before the full response is generated.
  • Barge-in: if the user starts talking, immediately stop TTS, flush the queued audio, and re-listen — interruption handling is core, not an edge case.
  • Tool calls happen mid-turn; fill dead air with a brief "let me check that."
Think firstWhich single number best predicts whether the agent feels responsive — total response time, or something else?Commit to an answer, then reveal

Time-to-first-audio. Users perceive the pause before the agent starts speaking, not how long the full answer takes — a response that begins in 400ms and talks for ten seconds feels snappy; one that starts after two seconds feels broken regardless of length. This is why every serious stack streams TTS by sentence and why vendors publish time-to-first-audio, not completion time. Optimize and monitor the start, not the total.

5. Evaluation (spine: evaluation)

  • Latency: p50/p95 turn latency and time-to-first-audio — the primary metric, broken down per stage (endpointing, ASR final, LLM first token, TTS first byte).
  • ASR quality: word error rate on your domain audio (accents, noise, jargon) — benchmark WER and production WER routinely diverge.
  • Task success: did the call accomplish its goal? (booking completed, question answered) — outcome eval on transcripts.
  • Conversation quality: interruption handling, naturalness, no talking over the user — often human-rated.
Deep dive
Generation & conversation quality metrics
Includes runnable notebook
Think firstASR might mis-hear a payment amount or an address. Confirm every detail with the user, then?Commit to an answer, then reveal

No — every confirmation costs a full slow turn and makes the agent feel robotic. The policy is selective: confirm only high-stakes, hard-to-reverse slots (payments, identity, addresses), triggered by ASR confidence, and let low-stakes details flow. Safety spent uniformly is latency wasted; spend it where an error actually hurts.

6. Guardrails & safety (spine: guardrails)

  • Same action guardrails as any tool-using agent: permissioned tools, HITL for risky actions, injection defense on transcribed content.
  • Voice-specific: graceful failure on ASR errors (confirm critical details: "did you say 3pm?"), and clear disclosure that it's an AI.
  • Latency vs. safety: confirmations add turns — reserve them for high-stakes slots (payment, identity), not everything.
Think firstTuesday's deploy doubled p95 turn latency. The pipeline has four stages plus network. What must already be in place for this to be a ten-minute investigation instead of a week?Commit to an answer, then reveal

Per-stage latency tracing on every turn: endpointing delay, ASR finalization, LLM time-to-first-token, TTS time-to-first-byte, and network, logged as a span per turn. Aggregate p95 says "slower"; only the stage breakdown says where — a regressed endpointer looks identical to a slow LLM in the end-to-end number. Voice is a distributed system with a 800ms SLO; you operate it like one.

7. Operations (spine: operations)

  • Serving: low-latency, high-concurrency inference — continuous batching, optimized ASR/TTS, and often speculative decoding to cut LLM latency.
  • Tracing: log audio, transcripts, and per-stage latencies per turn, plus tool calls — the per-stage breakdown is how you find the bottleneck.
  • Telephony: integrate with SIP/PSTN; handle jitter, packet loss, echo — and co-locate ASR/LLM/TTS regions, because a pipeline that hops clouds donates its budget to the network.
Deep dive
PagedAttention — efficient LLM memory for concurrent calls
Includes runnable notebook

8. Tradeoffs & alternatives

ChoiceAlternativeWhy
Streamed, overlapped pipelineSequential ASR→LLM→TTSSequential blows the latency budget
Cascaded ASR+LLM+TTSEnd-to-end speech-to-speech (Realtime API)Cascade is debuggable/controllable; s2s is lower-latency & more natural
Speak first tokens earlyWait for full responseTime-to-first-audio dominates perceived latency
Confirm only high-stakes slotsConfirm everythingEvery confirmation adds a slow turn

9. Interviewer follow-ups

The user interrupts mid-sentence. What happens?

Barge-in handling: VAD detects incoming speech, TTS stops immediately and flushes the queued audio, and the agent re-listens, incorporating the interruption. The context must also be truncated to what was actually spoken — the user is responding to the words they heard, not the paragraph you generated. Talking over the user is the most unnatural failure; interruption is a first-class path.

Where does the 800ms go, and how do you cut it?

Endpointing (waiting to be sure the turn ended), ASR finalization, LLM time-to-first-token, TTS time-to-first-audio, plus network per hop. Cut it by streaming partial transcripts, starting LLM reasoning before end-of-speech is certain, speaking the first sentence early, using fast TTS (~75ms-class models), and co-locating stages. Trace per-stage latency to find the real bottleneck — it's usually endpointing or the LLM, not TTS.

ASR mis-hears a critical detail (an address, a date). How do you handle it?

Confirm high-stakes slots explicitly ('I heard 742 Evergreen — correct?') using ASR confidence to decide when to confirm, and design tools to tolerate correction. Blindly acting on a low-confidence transcript for an irreversible action is the risk; targeted confirmation buys safety without slowing every turn.

Key takeaways

  • For a voice agent, latency is the architecture — target ~500ms p50 / 800ms p95 per turn, stream and overlap ASR, LLM, and TTS; time-to-first-audio is the metric that matters.
  • Barge-in, endpointing, and turn-taking are core interaction paths, not edge cases, and endpointing is the most underrated latency component.
  • Choose cascade for control (inspectable transcripts, per-stage swaps) or speech-to-speech for latency/naturalness; either way, add ASR-confidence confirmations for high-stakes slots and serve with continuous batching.

References