← Notes from the workbench

Why voice agents fail at turn-taking

Most voice agents don't fail at understanding — they fail at knowing when to speak. A look at endpointing, barge-in, and the awkward silence problem.

· 2 min read

Ask anyone who has shipped a voice agent what users complain about most, and it’s rarely comprehension. It’s rhythm. The agent interrupts mid-sentence, or leaves three seconds of dead air after the caller stops talking. Both failures come from the same place: turn-taking is a prediction problem, and most stacks treat it as a detection problem.

Detection vs. prediction

The default pipeline waits for silence. A VAD (voice activity detector) watches the audio stream, and when it sees N milliseconds without speech, it declares the turn over and lets the agent respond. Tune N low and you interrupt people who pause to think. Tune N high and every exchange gains a second of lag that makes the agent feel drugged.

Humans don’t work this way. We predict turn ends from prosody, grammar, and content — often before the last word lands. That’s why human conversational gaps average around 200ms, shorter than human reaction time. We’re not detecting the end of a turn; we’re forecasting it.

What actually helps

A few things that moved the needle for us in production:

  • Semantic endpointing. Feed the partial transcript to a small, fast model that scores “is this utterance complete?” A trailing “so…” or “my number is” should hold the floor even through long silence.
  • Two-stage response. Commit to a cheap acknowledgment early (“mm-hmm”, “got it”) while the full response generates. It buys 500ms of latency budget and sounds more human.
  • Barge-in as a first-class path. Users interrupt agents constantly. If your pipeline can’t cancel TTS mid-stream and re-enter listening in under 100ms, no amount of endpoint tuning saves you.
  • Measure gaps, not accuracy. Track floor-transfer time distributions in production. The p95 gap tells you more about perceived quality than word error rate ever will.

The uncomfortable truth

Turn-taking quality is mostly an infrastructure property, not a model property. A mediocre LLM on a pipeline with 300ms of end-to-end latency and proper barge-in will feel smarter than a frontier model that takes two seconds to answer. Latency is the personality of a voice agent — everything else is set dressing.