Twenty milliseconds: the unit that shapes realtime audio

If you build realtime voice systems, one number follows you everywhere: 20 milliseconds. It’s the frame size most telephony and streaming stacks settle on, and once you see why, a lot of otherwise-mysterious architecture decisions start making sense.

Why 20ms specifically

A frame is the atomic unit of audio your pipeline moves around. Smaller frames mean lower latency but more per-packet overhead; larger frames amortize overhead but add buffering delay. 20ms at 16kHz is 320 samples — small enough that no single stage adds perceptible lag, large enough that packet headers don’t dominate bandwidth. Opus, WebRTC, and most SIP deployments converged on it for good reason.

The consequence: your entire pipeline runs on a 20ms heartbeat. Fifty times a second, every stage — capture, VAD, STT, your orchestration layer — gets a tick. Miss ticks and you get jitter; jitter becomes either dropped audio or growing buffers, and growing buffers become latency users can feel.

Budgeting in frames

Thinking in frames turns “make it fast” into arithmetic. A rough budget for a sub-300ms voice loop looks like:

• Capture + network to your edge: 2–3 frames (40–60ms)
• Streaming STT emitting partials: 4–6 frames behind live audio
• Endpoint decision: 1–2 frames after the last word
• First LLM token: the scary one — often 10+ frames on its own
• TTS time-to-first-byte + playback start: 4–5 frames

Every stage that “just adds a little buffering” adds it in multiples of 20ms, and the multiples compound. The pipelines that feel instant are the ones where someone audited every stage and asked: how many frames are you holding, and why?

The practical takeaway

When a voice product feels sluggish, don’t start at the model. Start by tracing one frame through the system and counting where it waits. In my experience the model is responsible for less than half the perceived latency; the rest is queues nobody remembers adding. The 20ms frame is your unit of accountability — use it.