Realtime Voice AI Agent — ~1.3s Voice-to-Voice over WebSocket

Speak to it and it answers in about 1.3 seconds, end to end.

What it is

A real-time English voice agent: browser microphone in, streamed speech out, over a single WebSocket connection. Gemini Live handles speech recognition and reasoning; Azure neural TTS handles the voice. Measured round-trip latency sits around 1.3s from end of user speech to first audible syllable.

How the latency was won

• Nothing is buffered to completion. Audio streams in as chunks, model tokens stream out, and TTS synthesis starts on the first sentence boundary rather than waiting for the full response.
• The transport is one persistent WebSocket, not a sequence of HTTP requests — no repeated TLS handshakes or cold starts inside the conversational loop.
• Voice activity detection decides when the user has actually stopped talking instead of waiting on a fixed silence timeout, which is where most naive implementations lose half a second.
• Partial transcripts are used to warm the reasoning step before the utterance is finished.

Why it matters

Latency is the difference between a demo and something a person will actually talk to. Below roughly 1.5s a voice agent feels conversational; above 2.5s people start talking over it or hang up.

Stack

Gemini Live (STT + reasoning), Azure Neural TTS, WebSockets, FastAPI, Python, streaming audio pipeline.

Live demo, GitHub source and architecture: https://www.twine.net/signin