Home › Guides › Voice AI Agents
Tech Explained · 2026Voice AI Agents in 2026: How the Pipeline Works, Real Per-Minute Costs and 6 Steps to Ship One
Voice AI agents are software that listens to speech, decides what to do with a language model, and speaks back, usually in under a second. Production systems in 2026 split into cascaded STT-LLM-TTS pipelines and end-to-end speech-to-speech models, with all-in running costs typically landing between Rs 3.5 and Rs 22 a minute depending on which you pick.
- Latency is the product. 2026 production data puts end-to-end speech-to-speech p50 time-to-first-audio at 540 to 580ms and cascaded pipelines at 610 to 810ms. That 100ms gap decides fewer deployments than vendors claim.
- Cascaded still wins most briefs. You get a full transcript, component-level swaps, real tool calling and the freedom to put an Indic speech model in the STT slot. Speech-to-speech buys latency and charges you in lock-in.
- The LLM is your latency hog, not the speech models. In a typical 900ms budget, time-to-first-token eats roughly 400ms while TTS first audio costs 75 to 300ms.
- India has a genuine cost advantage. Sarvam's Samvaad voice agent platform lists Rs 3.5 per minute and its Saaras speech-to-text lists Rs 30 per hour of audio, well under dollar-denominated alternatives.
- Turn detection is where demos die. Pure voice activity detection talks over people. LiveKit's Turn Detector v1 and Pipecat's SmartTurnDetection exist because VAD alone is not enough.
- The skill transfers. Streaming, tool calling, latency budgets and barge-in handling are the same primitives behind every production agent, voice or not.
A caller says "actually, make that Thursday" halfway through your agent's confirmation sentence, and your agent keeps talking for another four seconds before noticing. By the time it stops, the caller has repeated themselves twice and hung up. Nothing in your logs looks broken. Every API returned 200. That single failure mode, not model quality, is what separates a voice agent demo from a voice agent that handles 12,000 calls a month.
What Is a Voice AI Agent?
A voice AI agent is a real-time loop: it takes a continuous audio stream, works out when the human has finished speaking, converts that speech to text, sends it to a language model that can call tools and query your systems, converts the reply back to audio, and plays it while still listening for the next interruption. The word that matters is loop. A text chatbot can take three seconds to think and nobody minds. A voice agent that takes three seconds has already lost the call.
What makes it an agent rather than an IVR menu is tool calling. The model does not just answer, it looks up your appointment table, reschedules the slot, and tells the caller what it did. If you have already built a RAG or agentic AI system, you know most of this stack; voice adds a hard real-time constraint on top of it and nothing else.
The cascaded voice agent loop
Six hops, one budget. Every arrow spends milliseconds you cannot get back.
Architecture as described by LiveKit and Pipecat documentation and vendor pipeline guides, checked 5 October 2026.
How Voice AI Agents Work: The Latency Budget You Are Actually Managing
Here is the thing most tutorials skip. You are not optimising a model, you are spending a budget. A caller's patience runs out somewhere under a second, and four components are competing for that second.
Vendor pipeline documentation puts the component costs at roughly 100 to 300ms for the first speech-to-text partial, 200 to 600ms for the language model's time to first token, and 75 to 300ms for the first audio chunk out of text-to-speech. Add network and jitter buffering and a well-built cascade answers in about a second. The useful insight: your language model is the expensive hop, by a wide margin.
Where a 900ms voice agent budget goes
Cut the LLM hop first. Swapping speech models buys you far less than you think.
Midpoints of the component ranges published in 2026 voice pipeline guides, normalised to a 900ms budget, checked 5 October 2026.
The three tricks that actually buy you time
Stream everything. Do not wait for a full transcript before you call the model, and do not wait for a full sentence before you start synthesising audio: send the first clause to text-to-speech while the model is still writing the second. Second, keep the system prompt short, because every token in it is prefill time before the first output token appears. Third, pick a smaller model for the conversational turns and escalate to a larger one only for the turns that need reasoning. Routing by turn type is the single biggest latency win available, and it is the same pattern we teach as part of tool-calling and streaming design in 360DT's live CCDV-F developer prep course.
Cascaded vs Speech-to-Speech: Which Voice Agent Architecture to Build On in 2026
This is the one real architectural fork, and the answer has flipped twice in eighteen months. End-to-end speech-to-speech models take audio in and emit audio out with no text in the middle. Cascaded pipelines chain three separate models. In 2026 production reports, speech-to-speech posted p50 time-to-first-audio of 540 to 580ms against 610 to 810ms for cascaded builds.
p50 time to first audio, by architecture
The headline gap is about 100ms. Humans leave roughly twice that between turns anyway.
Latency figures from 2026 voice agent benchmark write-ups and vendor pipeline documentation, checked 5 October 2026. Treat them as ranges, not guarantees.
| Criterion | Cascaded STT, LLM, TTS | End-to-end speech-to-speech |
|---|---|---|
| p50 time to first audio | 610 to 810ms in 2026 production reports | 540 to 580ms, the clear winner |
| Published per-minute price | Components priced separately, often cheaper in total | $0.05/min (Ultravox) up to $0.077/min speaking (gpt-realtime-2.1) |
| Tool calling maturity | Standard function calling, fully controllable | Supported, but tool latency shows through the audio turn |
| Full text transcript | Yes, by construction, for free | Only what the provider chooses to expose |
| Indian and code-mixed language quality | Swap in an Indic model: Saaras V4 covers 22 Indian languages plus English | Depends entirely on one frontier model's coverage |
| Interruption handling | You control it: cancel TTS, roll back the turn | Handled inside the model, and you cannot tune it |
| Debuggability when a call goes wrong | Per-stage traces tell you which hop failed | One opaque audio exchange |
| Emotion, prosody, laughter, tone | Weaker: text throws away how something was said | Noticeably better, this is the real reason to choose it |
| Telephony and SIP integration | Mature across every framework | Improving, still the rougher path |
| Data residency control | You choose each vendor and region independently | Whatever the single provider offers |
| Vendor lock-in | Low: replace any one stage in a day | High: the model is the architecture |
Choose cascaded if, choose speech-to-speech if
Choose cascaded if you need a transcript for compliance or quality scoring, if you serve Indian languages or code-mixed speech, if your agent calls more than two tools in a turn, or if you cannot accept one vendor owning your whole voice stack. That is most briefs in India.
Choose speech-to-speech if the emotional texture of the voice is the product, as in a language tutor, a companion app or a high-end concierge line, or if you have measured that 100ms and it genuinely changes your conversion. Mapping that kind of trade-off onto a deployable design is exactly the work the CCAR-F architect foundations track drills.
Our verdict: build cascaded unless prosody is the product. The latency gap is real but small, and you pay for it with lock-in, a missing transcript and no control over the one behaviour that breaks calls most often. Voice teams that go end-to-end first usually rebuild cascaded within six months, and they rebuild it under deadline pressure.
Voice AI Agents: Real Cost Per Minute in 2026
Pricing pages in this category are deliberately hard to compare, because some vendors bill audio tokens and some bill wall-clock minutes. Here is what the published figures said when we checked them on 5 October 2026.
| Option | What it charges | Per-minute equivalent | Worth knowing |
|---|---|---|---|
| OpenAI gpt-realtime-2.1 | $32 / $64 per 1M audio tokens in / out | ~$0.019 listening, ~$0.077 speaking | Token billing means a chatty agent costs more than a terse one |
| OpenAI gpt-realtime-2.1-mini | $10 / $20 per 1M audio tokens | Roughly a third of the above | The sane default for routine turns |
| OpenAI GPT-Live-1 | $0.05 per minute, full duplex | $0.05 | Released 10 September 2026; billed in concurrent sessions, 25 at Tier 1 to 500 at Tier 5, no free tier |
| AssemblyAI Voice Agent API | $4.50 per hour, flat | $0.075 | Flat-rate billing makes forecasting trivial |
| ElevenLabs Conversational AI | $0.08 per minute | $0.08 | Strongest voice quality in the bundled tier |
| Retell AI | $0.07 per minute | $0.07 | Telephony-first, built around phone numbers |
| Ultravox | $0.05 per minute, bundled speech-to-speech | $0.05 | Cheapest bundled end-to-end option we found |
| Sarvam Samvaad | Rs 3.5 per minute | Rs 3.5 | Indian-language voice agent platform, priced in rupees |
| Sarvam Saaras (STT only) | Rs 30 per hour of audio | Rs 0.50 | Rs 45 per hour with speaker diarisation |
| Sarvam Bulbul V3 (TTS only) | Rs 30 per 10,000 characters | Depends on reply length | Free tier of 1,000 credits for evaluation |
The number that matters is none of the above. Industry write-ups put most production voice agents at $0.10 to $0.25 a minute all in, because telephony, recording storage, observability and the inevitable retries land on top of model pricing. Budget from that figure, then work out which components you can pull below it. If you are routing between a cheap model and an expensive one per turn, an AI gateway is the cleanest place to put that logic, and the cost and latency instrumentation behind it is standard MLOps engineering work rather than anything voice-specific.
Voice AI Agents for Indian Languages: Where Indic Models Beat the Frontier Labs
This is where the generic advice stops being useful. A caller in Coimbatore does not speak Tamil or English; they speak both, in the same sentence, with an English noun and a Tamil verb. Frontier speech-to-speech models are weakest at exactly that, and you cannot fix it because there is no STT slot to replace.
Sarvam's Saaras V4 transcribes 22 Indian languages plus English and is built for code-mixed speech, while Bulbul V3 covers 11 Indian languages with more than 35 voices and handles Hinglish. Sarvam reached unicorn status in June 2026 after a $234 million Series B at a $1.5 billion valuation, which matters here only because it signals the Indic speech stack will still exist in three years. For teams that want the models inside their own VPC, the self-hosted route is viable for the language model even when speech stays managed. Azure's own speech services are the other common pairing, and wiring them into an agent is core AI-200 Azure AI developer territory.
A worked scenario: 14,000 minutes a month at a diagnostics chain
Take an illustrative case, not a real customer: a diagnostics chain in Coimbatore with a two-person engineering team wants outbound appointment reminders and rescheduling in Tamil and English. Twelve thousand calls a month, averaging 70 seconds, is 14,000 minutes.
On the platform route, Samvaad at Rs 3.5 a minute costs about Rs 49,000 a month and ships in two weeks. On the component route using the same vendor's building blocks, 14,000 minutes is 233 hours of audio at Rs 30 an hour, so roughly Rs 7,000 for transcription; at about 450 spoken characters per call, 5.4 million characters at Rs 30 per 10,000 is about Rs 16,200 for synthesis; add a few thousand rupees of language model spend and you are near Rs 27,000 before telephony, which you should price with your own provider rather than trusting anyone's blog.
So the platform costs roughly Rs 22,000 a month extra. For a two-person team, that is cheap: Rs 2.6 lakh a year buys you less than one engineer-month, and you will spend more than that building and babysitting the component pipeline. Our call is to start on the platform and move to components only when you cross about 40,000 minutes a month or when a compliance requirement forces you to own the audio. Volume is the trigger, not pride.
Voice agents are agents first. Learn the streaming, tool-calling and latency work that makes them hold up.
The AI Engineer course teaches generative AI, RAG and AI agents that plan, call tools and act, certified on both Microsoft Copilot Studio and Claude Code. You build and ship real agents live across 16 weeks, not toy notebooks.
Explore the course
Where Voice Agents Actually Work, and Where They Do Not
The deployments that succeed share a shape: high call volume, narrow intent, a clear escalation path, and a caller who already expected a phone call. Appointment reminders and rescheduling. Order status. Delivery slot changes. Loan document follow-ups. Outbound lead qualification, where recruitment teams report compressing a screening window from days to minutes. Clinic triage that collects symptoms and hands off. Analysing the resulting transcripts for intent and drop-off is ordinary SQL and Power BI analyst work, and it is usually where the first real ROI number comes from.
Now the honest caveat. Voice agents are oversold for anything with branching compliance scripts, anything where a wrong answer creates legal exposure, and anything a caller would rather do in an app. If your flow has nine decision points and a mandatory disclosure, a voice agent will either read a script badly or improvise something you will have to explain to a regulator. Build a form. Also: a meaningful share of Indian mobile calls happen on bad lines with a television on in the background, and accuracy on a noisy 8kHz phone codec is nothing like accuracy on your laptop microphone. Test on real calls from week one or your benchmark numbers mean nothing.
What Usually Goes Wrong: Turn Detection, Barge-In and Dead Air
Everyone builds the happy path in a weekend and then spends two months on turn-taking. Pure voice activity detection fires on silence, so your agent interrupts anyone who pauses to think, and waits politely through anyone who trails off. The industry's answer is a model that reads the transcript and predicts whether the thought is finished: LiveKit ships a Turn Detector built on a roughly 135M-parameter SmolLM-v2 fine-tune that runs locally and is free on its cloud inference, and Pipecat's SmartTurnDetection is reported to cut the agent talking over users by around 30% against pure VAD. Use one of them. Do not write your own silence threshold.
Barge-in is the other half. When the caller starts speaking while your agent is mid-sentence, the runtime has to cancel text-to-speech immediately, roll the interrupted turn back out of conversation history, and process the new input. Get the rollback wrong and the model replies to a question the caller abandoned. Get echo cancellation wrong and your agent interrupts itself, which sounds exactly as bad as it reads.
- Dead air with no fallback. If any hop stalls past about 1.5 seconds, play a filler ("let me check that") rather than silence. Silence makes callers hang up; filler buys you a second.
- Tool calls inside the voice turn. A 2 second database query is a 2 second pause. Acknowledge first, then query, then speak the result.
- Numbers and spellings. Phone numbers, order IDs and Indian surnames are where transcription accuracy collapses. Confirm them back digit by digit, always.
- No per-stage tracing. When a call goes wrong you need to know which hop was slow. Instrument every stage before launch, not after the first bad week.
- Testing on one accent. Your own voice is the least representative sample available to you.
Also read: What Is AI Agent Observability in 2026?
How to Build a Voice AI Agent: A 6-Week Plan That Ends in a Live Call
You do not need a research budget. You need a phone number, one narrow use case and the discipline to measure latency from day one.
Six weeks from empty repo to a live phone line
Each week ends in something you can demo on a real call.
Pick one intent and get audio flowing
Choose a single job: appointment rescheduling, order status, nothing broader. Stand up LiveKit Agents or Pipecat locally and get your own voice echoing back through the pipeline. Deliverable: a round trip you can hear.
Wire STT, the model and TTS with streaming on
Stream at every boundary. Send the first clause to text-to-speech before the model finishes. Deliverable: a working conversation with a logged time-to-first-audio number per turn.
Add real tool calls against a real system
Hook the agent to an actual database or booking API, not a mock. Add the acknowledge-then-query pattern so tool latency never becomes dead air. Deliverable: the agent changes something in your system during a call.
Fix turn detection and barge-in
Swap pure VAD for a semantic turn detector. Implement TTS cancellation and conversation rollback. Deliverable: you can interrupt the agent mid-sentence and it handles it cleanly.
Connect telephony and test on bad lines
Attach a SIP trunk or a provider phone number. Call it from a moving car, from a noisy room, in two languages. Deliverable: a scored transcript set of 50 real calls with word error rate per language.
Instrument, guardrail and ship to a slice
Per-stage tracing, a hard escalation path to a human, cost per call on a dashboard. Release to 5% of calls. Deliverable: a week of production traffic with latency and containment metrics.
Sequencing based on the component architecture and failure modes documented by voice agent frameworks, checked 5 October 2026.
Weeks 5 and 6 are where most side projects stop, and they are the only weeks a hiring manager cares about. Deploying WebRTC media servers, autoscaling on concurrent sessions and keeping p95 latency honest is cloud infrastructure work, which is why pairing voice building with a live AWS architect and DevOps programme tends to produce a far more employable portfolio than either skill alone. Handing the hard calls to a human is the same approval-queue design covered in our guide to human-in-the-loop AI agents.
Is Voice AI Worth Learning in 2026?
Yes, but not as a specialism. Treat voice as an interface you can add to agent skills you are building anyway. The transferable parts are streaming architecture, latency budgeting, tool-call orchestration and real-time failure handling, and those show up in every serious agent job description whether or not audio is involved. The genuinely voice-specific knowledge, VAD tuning, prosody, telephony codecs, is maybe three weeks of work.
What the market pays for is the combination. Indian voice AI platform companies hiring for this want someone who can do agent engineering and operate a real-time system, and industry reports suggest that combination is still scarce enough to command a premium over either half. If you want the vendor-verified version of the agent half, the generative AI developer track and the full certifications overview lay out which exams actually get read by recruiters.
Be sceptical of one claim you will hear a lot: that voice agents are about to replace contact centre headcount wholesale. What the adoption reporting actually shows is that around half of contact centres plan to adopt some form of AI, which is a very different statement. The jobs shift toward designing, auditing and escalating these systems. That is the role to aim at.
If you asked me what to do with the next six weeks: build one cascaded voice agent that does a single narrow job on a real phone number, in two languages, and measure its time to first audio on every turn. Skip the end-to-end model until something in your product genuinely needs the prosody, and skip the platform-versus-components argument until your volume makes it a real number. The reason that sequence works is that it forces you to learn streaming, tool orchestration and failure handling under a clock, which is the skill set behind every production agent rather than just the talking ones. If you want that built properly and reviewed live rather than guessed at from blog posts, the AI Engineer course is the straightest path to it.
Related guides
- AI Engineer Salary in India 2026 what the agent-building skills in this guide are actually worth, by experience and city.
- AI Engineer Interview Questions 2026 streaming, tool calling and latency come up in interviews exactly as they come up here.
- 8 Generative AI Project Ideas for 2026 if a voice agent is too large a first build, start with one of these.
- Highest Paying AI Jobs in India 2026 where real-time agent engineering sits against the other ten roles.
- What Is Context Engineering in 2026? the short-prompt discipline that keeps your time to first token low.
- What Is Apache Kafka? where call events and transcripts go once you are past a few thousand calls a day.
Frequently asked questions
Are voice AI agents worth building in 2026?
Yes, for high-volume narrow intents like appointment reminders, order status and lead qualification, where the caller expected a phone call anyway. They are a poor fit for branching compliance scripts or anything with legal exposure. Published platform pricing starts around Rs 3.5 a minute, and most production deployments land between $0.10 and $0.25 a minute all in.
What is the difference between cascaded and speech-to-speech voice agents?
Cascaded chains three models: speech to text, then a language model, then text to speech. Speech-to-speech is one model taking audio in and emitting audio out. Speech-to-speech is faster, at 540 to 580ms p50 against 610 to 810ms, and better at emotional tone. Cascaded gives you a transcript, component swaps, controllable interruption handling and the ability to use an Indic speech model.
How much does a voice AI agent cost per minute?
Published rates as of 5 October 2026 include Ultravox at $0.05 a minute, Retell AI at $0.07, AssemblyAI's Voice Agent API at $4.50 an hour which is $0.075, ElevenLabs Conversational AI at $0.08, and Sarvam Samvaad at Rs 3.5. Industry write-ups put all-in production cost at $0.10 to $0.25 a minute once telephony, storage and observability are included.
What latency do voice agents need to feel natural?
Aim for first audio back inside 800ms, and under 600ms if you can. Component budgets published in 2026 pipeline guides allow 100 to 300ms for the first speech-to-text partial, 200 to 600ms for the model's time to first token, and 75 to 300ms for the first audio chunk. The language model is almost always the hop worth optimising first.
Which voice AI framework should I learn, LiveKit Agents or Pipecat?
LiveKit couples the agent framework to its own WebRTC media server and cloud, and ships a built-in turn detector based on a roughly 135M-parameter SmolLM-v2 fine-tune. Pipecat is transport-agnostic and lets you bring your own media layer. Learn LiveKit if you want one vendor for media and agents, Pipecat if you want to choose your transport. The pipeline concepts transfer either way.
Can voice AI agents handle Hindi, Tamil and code-mixed Indian speech?
Through a cascaded pipeline, yes. Sarvam's Saaras V4 transcribes 22 Indian languages plus English and is built for code-mixed speech, and Bulbul V3 synthesises 11 Indian languages with more than 35 voices including Hinglish. End-to-end speech-to-speech models from frontier labs are generally weaker on code-mixed Indian speech, and you cannot swap in a better component because there is no component to swap.
What breaks most often in production voice agents?
Turn detection and barge-in. Pure voice activity detection interrupts callers who pause and waits through callers who trail off, which is why LiveKit and Pipecat both ship semantic turn detectors. After that: dead air during tool calls, transcription errors on phone numbers and Indian surnames, and no per-stage tracing to tell you which hop was slow.
Do I need a machine learning background to build voice AI agents?
No. You are calling hosted speech and language models, not training them. What you need is comfort with real-time systems: streaming, cancellation, concurrency and latency measurement. If you can build a RAG application and reason about p95 latency, you can build a voice agent in about six weeks.
About this guide. 360 Digital Transformation is an Anthropic Partner. Other certification bodies, vendors and employers named here are not affiliated with us. Product features and pricing change often; figures cited were checked on 5 October 2026.
Python
Claude API
FAISS
ChromaDB
LangChain
LangGraph
CrewAI
AutoGen (AG2)
