Case study · MILL5 · 2024 — 2026
Real-time conversational voice agents
Streaming speech recognition, LLM orchestration and speech synthesis tuned for natural turn-taking.
Staff AI Engineer — architecture and latency work
- Reduction in average interaction latency
- 45%Reduction in average interaction latency
- Faster information processing with speech AI at WellSky
- 30%Faster information processing with speech AI at WellSky
Context
Voice is unforgiving. In a text interface a two-second pause is fine; on a phone call it feels like the line has dropped. The first versions of the agents answered correctly, but too slowly to feel like a conversation.
Approach
- Treated the pipeline as streaming end to end: partial transcripts from ASR, streamed LLM tokens, and speech synthesis that starts on the first complete phrase.
- Tuned voice activity detection and endpointing so the agent stops talking when interrupted and doesn’t cut people off mid-sentence.
- Set a latency budget per stage and traced every turn, so regressions pointed at a specific component.
- Integrated telephony through Twilio Voice and browser audio over WebRTC.
Results
- Average voice interaction latency dropped by 45%.
- Per-stage tracing made it straightforward to evaluate new speech providers against the existing budget.
What I took from it
Perceived latency matters more than measured latency. Starting to speak sooner — even with a short acknowledgement — did as much for the experience as making any single model faster.
Stack
- Whisper
- Deepgram
- ElevenLabs
- Twilio Voice
- WebRTC
- VAD
- WebSockets