How We Built a Real-Time AI Voice Agent: The Full Architecture

Introduction

A real-time voice agent has to do a lot of small things in order, and it has to do them fast enough that the person on the other end feels like they are talking to something that is actually listening. It needs to notice when someone starts speaking, turn that speech into text, decide when the person has finished their thought, send it to a language model, and then speak the answer back. All of that has to happen while audio keeps streaming in both directions.

This is a write-up of how we built one. It is a real, production-grade voice agent that runs on the web and over the phone, and we built it in about a quarter. I am keeping some specifics general on purpose, like exact timing values and the exact vendors we chose, but the architecture itself is the interesting part and that is what I want to walk through.

What you'll learn:

• The full real-time loop, step by step, from a person speaking to the agent speaking back
• How the agent decides when your turn has ended, and how it handles you cutting it off
• The two latency numbers we watch, and the startup trick that shaved time off every call
• Why providers are swappable per agent, and how grounding and reach are wired in

The loop

The whole system is one loop that repeats for every turn in a conversation. It is built on an open-source framework for voice agents called LiveKit Agents, which handles the plumbing of moving audio around in real time so we could focus on the parts that matter to us.

Here is the loop in plain terms. First the agent listens to the incoming audio and decides whether anyone is actually speaking. That step is called voice activity detection, and for it we use a model called Silero. It is a small, fast check that answers one question many times a second. Is this sound speech, or is it silence and background noise? We only want to act on real speech, so this gate sits at the very front.

Once it hears speech, the audio goes to speech to text, which turns the spoken words into written text as the person talks. That text is what the rest of the system reasons about. The text goes to a language model, which is the part that decides what to say. The model's answer then goes to text to speech, which turns the written reply back into spoken audio. That audio streams back to the person.

The word that matters across all of this is streaming. We do not wait for the person to finish, then wait for the full transcript, then wait for the full answer, then play it. Each stage starts working on partial output from the stage before it, so the parts overlap instead of stacking up. The single most important idea in a real-time voice agent is that every stage streams into the next one, because waiting for each step to fully finish is what makes an agent feel slow and robotic.

Turn-taking and interruption

The hardest part of a natural conversation is not the words. It is knowing when it is your turn to talk. People do not announce that they are done. They trail off, they pause in the middle of a sentence to think, and they expect the other side to read it correctly.

A simple silence timer is not good enough for this. If you wait for a fixed gap of quiet, you either cut people off when they pause to think, or you leave long awkward delays after they are clearly finished. So instead of a timer, we use a model whose only job is turn detection. It looks at what the person has said and how they said it, and it decides whether they have actually finished their turn or are just pausing. That lets the agent jump in quickly when someone is done, and wait patiently when someone is mid-thought.

The other half of natural conversation is interruption. If the agent is talking and you start to speak, it needs to stop and listen, the same way a person would. That is called barge-in. When the agent detects that you have started speaking over it, it stops its own speech and hands the floor back to you. Without barge-in, the agent talks over people and the conversation falls apart. I am keeping the exact timing values we tuned to ourselves, but the idea is what counts: turn-taking is a model decision, not a fixed clock, and the agent always yields when interrupted.

The latency budget

In a voice conversation, delay is the thing people feel first. If the agent takes too long to start answering, it feels broken even when the answer is good. So we measure latency on every single turn, end to end, and we treat it as a number we are responsible for rather than something we hope is fine.

We watch two numbers in particular. The first is time to first token, which is how long it takes the language model to start producing its answer after it receives the person's words. The second is time to first audio, which is how long it takes for the person to actually hear the agent start speaking. The first one tells us the model is thinking, the second tells us the person is hearing something. Both matter, and they are not the same.

A lot of the work was in the startup path, the moment a call connects and the agent has to be ready. One example I can share is that we pass the agent its configuration before it connects, rather than having it connect first and then go fetch what it needs. That small reordering means the agent is not sitting idle waiting for setup while the person waits for it. Latency is not one big problem you solve once. It adds up across many small steps, and the startup path is where a surprising amount of it hides. I am keeping the exact millisecond targets private, but the practice is the point: measure every turn, and attack the slowest step.

Swappable providers

We do not hard-wire the agent to one set of vendors. For the three external pieces, speech to text, the language model, and text to speech, we use strong third-party providers, and each one can be swapped per agent. One agent can use a different model or a different voice than another, and we change that through configuration rather than through code.

There are two reasons this matters. The first is that this space moves fast, and a provider that is best today may not be best in six months, so we did not want to be locked in. The second is reliability. If a provider is not configured or is unavailable, the agent automatically falls back to another option rather than failing the call. Treating each provider as a swappable part, with automatic fallback, is what lets a voice agent survive a moving market and an outage without a rewrite. If you want a broader view of how voice agents are being built right now, I wrote a fuller guide here: the 2026 guide to voice AI agents. And if you are curious about the speech to text layer specifically, I went deep on one option here: ElevenLabs Scribe v2 for real-time speech to text.

Grounding

A voice agent that makes things up is worse than no agent, because people trust a confident voice. So the agent does not answer from the language model's memory alone. Each agent has its own knowledge base, and it answers from that.

The way it finds the right information is vector search. In plain words, every piece of the knowledge base is turned into a list of numbers that captures its meaning, and so is the person's question. The agent then finds the pieces whose meaning is closest to the question, and uses those to answer. We run this on MongoDB Atlas Vector Search. The benefit is that the agent answers from real, owned content rather than guessing.

There is one more layer. When the agent's confidence in what it found is low, it falls back to a web search so it can still try to give a useful answer instead of a wrong one. Grounding the agent in a per-agent knowledge base, with a web-search fallback when confidence is low, is the difference between an agent that answers and an agent that invents. I am keeping the embedding details and the web-search vendor private, but the shape is what matters.

Reach over web and phone

The last piece is reach. The same agent works in two places without us building it twice. On the web it runs in a small embeddable widget, so it can live on a page. Over the phone it answers real calls, which we route through Twilio using LiveKit's phone connection layer, called SIP. The agent behind both is the same agent, so its knowledge and its behavior stay consistent whether someone types on a website or dials a number. There is also an optional video avatar for cases where a face on screen helps.

Underneath all of this sits a four-part system: a dashboard and the embeddable widget that people use, a control plane that manages the agents, the real-time worker that runs the loop I described above, and a shared contracts package that keeps the other three working from the same definitions. One agent that reaches people over the web and the phone, from a single definition, is what makes the whole thing worth building rather than a demo.

Conclusion

None of these parts is exotic on its own. The work was in wiring them into one loop that stays fast, decides turns like a person, grounds its answers, and reaches people wherever they are. We built it in about a quarter as a real production system, and the lesson I keep coming back to is that a good voice agent is mostly an exercise in latency and turn-taking, not in any single clever model. If you want the bigger picture of how I build these systems with a team of agents, I wrote about that here: from prompts to an AI engineering team.

Further Reading

• The 2026 guide to voice AI agents
• Beating LLM hallucinations with RAG
• Production-ready AI agent architecture
• From prompts to an AI engineering team