Skip to content

Case study · MILL5 · 2024 — 2026

Real-time conversational voice agents

Streaming speech recognition, LLM orchestration and speech synthesis tuned for natural turn-taking.

Staff AI Engineer — architecture and latency work

Reduction in average interaction latency
45%Reduction in average interaction latency
Faster information processing with speech AI at WellSky
30%Faster information processing with speech AI at WellSky

Context

Voice is unforgiving. In a text interface a two-second pause is fine; on a phone call it feels like the line has dropped. The first versions of the agents answered correctly, but too slowly to feel like a conversation.

Approach

  • Treated the pipeline as streaming end to end: partial transcripts from ASR, streamed LLM tokens, and speech synthesis that starts on the first complete phrase.
  • Tuned voice activity detection and endpointing so the agent stops talking when interrupted and doesn’t cut people off mid-sentence.
  • Set a latency budget per stage and traced every turn, so regressions pointed at a specific component.
  • Integrated telephony through Twilio Voice and browser audio over WebRTC.

Results

  • Average voice interaction latency dropped by 45%.
  • Per-stage tracing made it straightforward to evaluate new speech providers against the existing budget.

What I took from it

Perceived latency matters more than measured latency. Starting to speak sooner — even with a short acknowledgement — did as much for the experience as making any single model faster.

Stack

  • Whisper
  • Deepgram
  • ElevenLabs
  • Twilio Voice
  • WebRTC
  • VAD
  • WebSockets