Building a voice-first AI system that answers phone calls in real-time is harder than it looks. Here's what we learned shipping Autonomy Receptionist.

The Challenge

When someone calls a business, they expect an answer in under 3 seconds. That means our entire stack—from Twilio webhook to LLM response to voice synthesis—needs to complete in under 2 seconds to feel natural.

Traditional chatbot architectures don't work here. You can't afford the luxury of:

Sequential API calls
Waiting for full responses before speaking
Retries on failure

Our Architecture

We built a streaming pipeline that handles voice in real-time:

1. Voice Input (Twilio → Deepgram)

// Twilio forwards audio stream via WebSocket
twilioStream.on('media', async (media) => {
  // Stream directly to Deepgram for real-time transcription
  const transcript = await deepgram.transcribe(media.payload);

  // Process while still listening
  processTranscript(transcript);
});

2. Intent Detection (Streaming LLM)

Instead of waiting for the full transcription, we use GPT-4's streaming API to start generating responses as soon as we have partial context.

const response = await openai.chat.completions.create({
  model: "gpt-4",
  messages: conversationHistory,
  stream: true, // Critical for low latency
});

for await (const chunk of response) {
  // Start synthesis immediately
  synthesizeAndPlay(chunk.choices[0]?.delta?.content);
}

3. Voice Synthesis (Parallel Processing)

We don't wait for the LLM to finish before speaking. As soon as we have enough tokens for a complete sentence, we send it to voice synthesis.

Key Optimizations

1. Warm Connections We keep persistent connections to OpenAI and voice synthesis APIs. Cold starts are the enemy of real-time.

2. Predictive Responses For common queries ("What are your hours?"), we pre-generate responses and cache them. This cuts latency from 1.5s to 200ms.

3. Graceful Degradation If the LLM is slow, we use canned responses while waiting. Better to sound robotic than to have dead air.

Lessons Learned

Streaming is non-negotiable. Sequential processing adds 500ms+ per step. Stream everything.

Latency compounds. A 100ms delay in transcription becomes 500ms by the time it reaches the user. Optimize ruthlessly.

Cache aggressively. Most business calls follow patterns. Don't recompute the same responses.

Results

Under 2s average response time
97% transcription accuracy (Deepgram)
$0.12 per minute (all-in cost)

Voice AI works when you treat it like infrastructure, not magic.

Want to see it in action? Check out Autonomy Receptionist.