How Voice Agents Handle Latency, Turn-Taking, and Safety

Hansen describes a voice agent as a conversation with a machine using speech. Speech-to-text converts the user's words to text, an LLM processes them and generates a response, and text-to-speech turns that response back into audio. He says ElevenLabs can also convey expression such as laughter or whispering.

Hansen says the LLM may account for 40 to 70 percent of pipeline latency, and that faster settings can reduce expressiveness or correctness. Audio cues, such as keyboard sounds while an agent looks something up, can signal that work is happening. Turn-taking systems can distinguish acknowledgments from interruptions, while safety checks can run in streaming mode or verify output before it is spoken, with different latency tradeoffs.

A simple task such as routing a caller can use a faster model, then hand off to a more capable sub-agent for a complex task like loan advice. Teams can choose whether the voice changes during the transfer or stays the same. Hansen says the choice can help match the agent's tone to the interaction.

Hansen describes a universal JavaScript library, a React package with hooks and providers, and a React Native package that adds platform-specific audio input. The layered design builds framework and platform features on a shared core. Developers can bring their own UI, use provided components, or embed a chat bubble.

Hansen says ElevenLabs offers both an integrated Agents platform and services that let developers bring their own LLM, including Speech Engine for adding voice to an existing chat experience. He wants to generate specifications from existing SDK implementations, compare them across languages and platforms, and bring their experiences closer together.