Voice AI latency: why it makes or breaks phone voice agents

The physics of feeling human

In human conversation, the gap between turns averages a couple hundred milliseconds. On the phone, silence after a question stretches subjective time: at one second the caller notices, at two they suspect the line dropped, at three they start talking, directly over the agent's late reply, derailing the turn entirely. A voice agent that answers brilliantly in four seconds loses to one that answers adequately in one and a half.

Where the milliseconds go

The response chain has four stages. End-of-turn detection decides the caller has finished, naive silence thresholds burn 500–1000 ms here, while semantic approaches that judge whether the utterance is linguistically complete cut it sharply. Recognition finalization commits the last words. Reasoning generates the reply, where model choice and prompt size dominate. Speech synthesis produces the first audio byte, and synthesis that waits for the full reply before speaking is the single most common latency mistake.

Everything must stream. The model starts generating before recognition would have fully settled in a batch world; synthesis speaks the first sentence while the second is still being written; audio flows to the caller as it is produced. A pipeline that runs its stages sequentially cannot hit conversational budgets no matter how fast each stage is.

Engineering perception, not just totals

Production systems also shape how latency feels. A brief natural acknowledgment, rendered instantly from cache while the real answer is prepared, fills the gap the way a human's 'let me check that' does. Sentence-level pipelining keeps audio flowing continuously once speech starts. And per-turn telemetry across recognition, reasoning, and synthesis makes regressions visible the day they ship, not the week complaints arrive.

A worked latency budget

It helps to put numbers on the chain. Target first audio within 1,500 ms of the caller falling silent, and the budget divides roughly like this. Semantic end-of-turn detection: 100–300 ms, against 500–1,000 ms for a naive silence timer, which is why this is the first place to look. Recognition finalisation of the trailing words: 50–150 ms if recognition has been streaming throughout, far more if it was batched. Any tool call the turn requires, a CRM lookup, an order status, an availability check: 200–600 ms, and this is the stage most teams forget to budget for at all. Reasoning to the first token: 200–500 ms. First synthesised audio byte: 100–300 ms with streaming synthesis.

Add the middle of those ranges and a turn with a tool call lands near 1.4 seconds, with no headroom. That arithmetic explains two things at once: why a system feels fine in a scripted demo and slow in production, where real turns hit real systems, and why shaving 400 ms off end-of-turn detection changes the character of the whole conversation.

Note what is not in the budget. Network transit to a distant region is pure additive cost on every stage that crosses it, which is why in-region deployment is a latency decision before it is a compliance one. A round trip from the Gulf to a European or US region can add 120–250 ms per hop, and a pipeline that crosses the water three times has spent a third of its budget on geography.

Why telephony and Arabic both make it harder

Phone calls are not web calls. Carrier audio arrives narrowband, typically 8 kHz, which strips the high-frequency information recognisers lean on, so recognition needs more context before it commits, which costs time. Jitter buffers add their own delay to smooth packet arrival. And unlike a browser demo, there is no visual channel: the caller has no spinner, no typing indicator, nothing at all to tell them the system is working. Silence is the only signal they get, which is exactly why the same total latency feels worse on a phone than in a chat widget.

Arabic adds two specific pressures. Code-switching mid-sentence, which Gulf callers do constantly, forces either a multilingual recogniser or a language-detection step, and a detection step in the hot path is a latency stage. And dialectal speech carries more recognition uncertainty than newsreader Arabic, which tempts a system into waiting for more audio before finalising. Both push in the same direction: toward buffering, and buffering is latency.

The mitigation is the same in both cases, which is to move the uncertainty out of the hot path. A recogniser that handles Arabic and English in one model removes the detection stage entirely. Dialect-aware acoustic modelling reduces the confidence deficit that causes waiting. Neither is a latency optimisation on the surface; both buy hundreds of milliseconds.

Measure p95, not the average

Averages are actively misleading for conversational latency, because callers do not experience an average, they experience the worst turn in their call. A system with a 900 ms median and a 4-second 95th percentile will read as fast on a dashboard and feel broken to roughly one caller in twenty, and those are the calls that end in a hang-up or a complaint. Report the median for tuning and the 95th percentile for truth.

Instrument per stage, per turn, not per call. A single end-to-end number tells you a turn was slow and nothing about which stage caused it, and the stages fail for unrelated reasons: end-of-turn regressions come from VAD tuning, reasoning regressions from prompt growth, synthesis regressions from a cold voice cache, tool-call regressions from somebody else's API. Stamp each stage and the slow turns diagnose themselves.

Two further habits are worth building in. Record whether a turn involved a tool call, because mixing tool turns and conversational turns into one distribution hides both. And alert on the drift, not just the threshold: a pipeline that moves from 1.1 to 1.6 seconds over a month never trips a two-second alarm but has already lost the room.

Frequently asked questions