Callie — Real-Time AI Phone Receptionist (Twilio + Gemini Live, barge-in handling)

An AI receptionist that answers real phone calls, gets interrupted mid-sentence, and handles it.

What it does

A caller dials a normal phone number. Twilio media streams are bridged in real time to Gemini Live, which listens, reasons and speaks back. The agent books callbacks, opens tickets, answers questions from the business knowledge base, and writes a post-call summary into the backend.

The hard parts

• Barge-in. Humans interrupt. The agent has to stop talking mid-word, flush queued audio, re-open the input stream and re-anchor on what was actually said — not what it was about to say. Getting this wrong makes a voice agent unusable, and it is most of the engineering.
• Latency budget. Telephony audio in, model reasoning, speech out, all inside the window where a caller does not think the line dropped. That means streaming everything, never buffering a full turn, and cutting round trips.
• Telephony reality. 8kHz mu-law audio, codec conversion, dropped packets, DTMF, call transfer and hangup edge cases.
• Actions with consequences. Tool calls that create tickets or schedule callbacks are validated and logged, not fired blindly on a transcript the model may have misheard.

Stack

Twilio Media Streams, Gemini Live, ElevenLabs, WebSockets, FastAPI, Celery, PostgreSQL.

Live demo, write-up and architecture: https://www.twine.net/signin