Solving the Latency Problem in Voice AI Agents

Why latency matters so much in voice

People take turns in conversation fast. A study of turn-taking across ten languages (Stivers et al., PNAS, 2009) found the usual gap between speakers is around 200 ms. A pause much longer than a second sounds like hesitation, confusion, or a dropped line.

A chat window can show a typing indicator while the system thinks. A phone call has nothing to fill the gap, so the caller hears silence and reacts to it. They repeat the question, start talking just as the agent answers, or hang up.

The delay also adds up over a call. With 2 seconds of silence on each of 20 turns, a support call carries 40 seconds of dead air.

Most teams we work with aim for under about 800 ms from the end of the caller's speech to the start of the agent's reply, and treat under 500 ms as excellent. These are working rules of thumb rather than published standards, so test them with your own callers.

Where the milliseconds go

A single reply goes through six stages, and the caller hears the total. The table shows a rough budget for a well-tuned pipeline. Your numbers will depend on region, network, and the models you pick.

Turn detection and the LLM usually take the most time. If an agent waits out a fixed 700 ms of silence and then calls a large model, one turn can easily run past 1.5 seconds.

Note that vendor latency figures typically cover model inference only. ElevenLabs, for example, quotes Flash v2.5 at a median of about 100 ms, excluding application and network latency (ElevenLabs docs). End-to-end measurements are always higher, so budget for the whole chain.

What slow agents cost

Slow replies show up in containment rate, customer satisfaction, and cost per call.

Callers who hear dead air ask for a human or hang up, and each escalation eats into the savings that justified the agent. Slow agents also get talked over. Callers repeat themselves, the agent hears two questions at once, misreads the intent, and the call drags on.

Longer calls cost more, because platform and telephony fees are charged per minute. A hesitant voice also sounds unreliable, which hurts conversion in sales and booking calls. And pilots get judged on how a demo call feels, so a laggy pilot rarely makes it to production even when its answers are accurate.

A fast agent can be trusted with harder, higher-value calls. Track containment, CSAT, and average handle time before and after latency work so you can see the effect on your own traffic.

How to cut latency

The biggest gains come from running stages in parallel instead of shaving time off one of them.

1. 01Stream every stage. Use streaming speech-to-text, stream tokens out of the LLM, and send the first phrase to text-to-speech before the full reply is written. The caller hears the start of the answer while the rest is still being generated.
2. 02Replace silence timeouts with a turn-taking model. A model that listens to the words and the audio can tell "I need to... um" from a finished sentence, so the agent answers sooner without cutting people off.
3. 03Start the reply early. Speculative generation begins drafting a response during a pause and throws it away if the caller keeps talking.
4. 04Use the smallest model that does the job. Send only the hard turns to a larger model, keep system prompts short, and cache context that doesn't change.
5. 05Cover the waits you can't remove. When a tool call is slow, have the agent say something like "Let me pull up your booking" while the request runs.
6. 06Keep connections open. A persistent WebSocket avoids a new handshake each turn, which can cost 100 to 250 ms.
7. 07Host near your callers. Run the agent and models in the region the calls come from, since each long network hop adds tens of milliseconds.
8. 08Stop talking when interrupted. Cut the agent's audio as soon as the caller speaks.

You will have to trade some quality for speed. The most expressive voices and the largest models are rarely the fastest, so choose per use case.

Where ElevenLabs fits

ElevenLabs works on the two slowest parts of the chain, speech output and turn-taking, and puts them in one agent platform.

All ElevenLabs latency figures are medians of model time only, excluding application and network latency.

ElevenAgents lets you choose your own LLM, so your end-to-end latency depends a lot on that choice, and ElevenLabs says as much. Measure it on your own setup before you commit to a target.

For real-time calls, Eleven v4 Turbo and Flash v2.5 are the options to look at. The flagship Eleven v4 model sounds richest but suits recorded content better than live calls. Flash v2.5 doesn't format numbers, dates, or currencies for speech by default, so have the LLM write them out first. As an ElevenLabs partner, we help clients choose the right mix for each use case.

Measuring latency

Measure what the caller hears, not the figures vendors publish. Track these per turn and report the 50th, 90th, and 99th percentiles, because a handful of slow turns can spoil a call even when the average looks fine.

Treat these targets as starting points. Test from the networks your callers actually use, including mobile and phone lines, and test again whenever you change a model or prompt.

A rollout plan

We run latency work in four phases and move on only when the measurements, not the demo, say a phase is done.

1. 01Baseline (weeks 1 to 2). Instrument every stage, record real calls, and set p50 and p90 targets for voice-to-voice latency.
2. 02Quick wins (weeks 2 to 4). Stream everything, pick a low-latency voice model such as Eleven v4 Turbo or Flash v2.5, turn on the turn-taking model, and keep connections open.
3. 03The language model (weeks 4 to 8). Benchmark LLMs on your own tasks, shorten prompts, add filler lines for slow tools, and route complex turns separately.
4. 04Scale and monitor (ongoing). Deploy in your callers' regions, load-test at peak volume, and set alerts on p90 latency.

Pre-launch checklist

• Voice-to-voice latency measured from real caller networks
• p90 latency within target under peak load
• Turn-taking tested with hesitant, fast, and noisy speakers
• Barge-in stops agent audio cleanly
• Slow tool calls covered by bridge phrases
• Latency dashboards and alerts live

The weeks above are typical. Your timeline depends on scope and how many systems the agent connects to.

Next steps

ElevenLabs gives you fast speech models and a turn-taking model to build on. The rest of the gain comes from how you stream the pipeline, which LLM you pick, and measuring on real calls. Start with the baseline phase above and you'll know within two weeks where your own delay sits.

About Haass

Haass is an ElevenLabs partner that designs, builds, and optimizes voice AI agents for companies in any industry. To discuss a latency audit or a voice agent pilot, contact us at marketing@haass.io.

Sources

• ElevenLabs: Models documentation
• ElevenLabs: Meet Flash (original 2024 launch post)
• ElevenLabs: Interaction models for natural human-AI communication
• ElevenLabs: ElevenLabs Agents vs OpenAI Realtime API
• Stivers, T. et al. (2009). Universals and cultural variation in turn-taking in conversation. PNAS 106(26).