Building Real-Time Voice AI Agents with Vapi and Deepgram

Voice agents feel like magic when they work and unbearable when they lag. The difference is almost always latency budget, the total time from the moment a user stops speaking to the moment they hear a reply. Get it under ~800ms and the conversation feels human; cross 1.5s and people start talking over the bot.

The three-stage pipeline

Every real-time voice agent is the same pipeline: speech-to-text (STT), a language model that decides what to say, and text-to-speech (TTS) that speaks it back. The art is in overlapping these stages instead of running them one after another.

• STT, Deepgram streams partial transcripts as the user speaks, so you are not waiting for a final transcript to start thinking.
• LLM, stream tokens out as they generate, and begin synthesizing speech from the first sentence rather than the full response.
• TTS, Deepgram or ElevenLabs turns those streamed sentences into audio chunks the moment they are ready.

Where Vapi fits

You can wire this yourself with websockets, a turn-taking state machine, and careful barge-in handling, I have. But Vapi orchestrates the STT → LLM → TTS loop, handles interruptions, and gives you a clean webhook surface for tool calls. It turns a multi-week build into a configuration problem, which is exactly what a client wants to pay for.

// A tool the voice agent can call mid-conversation
async function bookAppointment(args: { date: string; service: string }) {
  const slot = await db.slot.findFirst({
    where: { date: args.date, available: true },
  });
  if (!slot) return { ok: false, message: "That slot is taken." };
  await db.slot.update({ where: { id: slot.id }, data: { available: false } });
  return { ok: true, message: "Booked. You'll get a confirmation text." };
}

The lesson

Treat latency as a first-class requirement, stream everything, and let a platform like Vapi handle the orchestration so you can spend your time on the part that is actually yours: the tools the agent can call and the business logic behind them.