Real-Time Phone Voice Agent: Fixing Silence and Latency

We built a voice agent that answers real phone calls.

Each call opened with silence while the speech services and language model started. The greeting never changes, so we generate its audio once, when the program starts. We store that audio in memory as 20 ms chunks, ready to send when the call connects.

A second bug silenced every call after the first. Some speech services store session state: data tied to one call. Reusing them breaks the next call. Only the language model is safe to reuse between calls.