Voice agents split into three jobs: the brain, the ears and the gatekeeper

The brain: context and governance

Databricks argues that voice agents deliver real customer value when enterprise context, ontology and governance sit behind the conversation. In its architecture, Databricks provides the brain, handling context, ontology and governance, while specialist partners such as LiveKit, Deepgram and ElevenLabs provide the conversation layer for audio and speech-to-text. The split is deliberate: the company that owns the customer data and the rules should own the decisions, while the audio layer can be swapped. m:…8790

Databricks uses System 1 fast decision models and System 2 large language models to handle different types of call reasoning, and its ai_decide function allows agents to make quick decisions based on governed data. Agents follow existing enterprise data catalog access rules, every decision made by a voice agent on Databricks is recorded for review, and sensitive topics like medical or financial questions are routed to humans by design. The company supports HIPAA and SOC 2 Type II compliance for voice agent data, and it recommends a 90-day pilot for the first voice agent implementation, with its DIVA blueprint available as an open-source starting point on GitHub. m:…8790

Gartner predictions cited by Databricks put 80 percent of common customer service issues resolved by agentic AI by 2029, with a 30 percent reduction in operational costs from agentic AI by 2029. Those are forecasts, not results, but they explain why enterprises are piloting voice agents now: the cost case is being made on a multi-year horizon, and the governance work done in the pilot will decide whether the forecast is reachable. m:…8790

The ears: hearing more than one speaker

Inflection AI is upgrading its conversational AI, Pi, to handle complex multi-speaker interactions by integrating NVIDIA Nemotron Speech open models into its Conversational Interaction Stack. The team implemented NVIDIA Streaming Sortformer for speaker diarization, integrated NVIDIA Multitalker-Parakeet for speaker-conditioned speech recognition, and integrated NVIDIA Nemotron 3 Diarization to improve diarization accuracy. Multitalker-Parakeet enables separate transcription of overlapping speech, which is the hard part of group conversation: people talk over each other, and a transcript that merges them is worse than useless. m:…5187

Inflection is also testing NVIDIA Nemotron 3.5 Transcribe for multilingual support, developing buffering and transcript-handling logic for it, and evaluating inference weight and latency for the new speech recognition pipeline. The stated aim is to make conversations with Pi feel natural and seamless, and the engineering detail that matters is latency: a multi-speaker system that transcribes accurately but slowly will still feel broken in a live conversation. m:…5187

Baseten attacks the same hearing problem from the cost side. Its optimized stack for the Qwen3-ASR 1.7B model, an open-weight speech recognition model released under an Apache 2.0 license, supports more than 30 languages and 22 Chinese dialects with a 5.4 percent average word error rate. Baseten reports 7 times more audio transcribed per second at high concurrency compared with vanilla vLLM deployments, GPU utilization rising from 86 percent to 97 percent by minimizing host-to-device syncs, and a sustained factor of over 3,000 times real-time for hour-long files. m:…9908

Baseten reports that the optimized deployment transcribes an hour of audio for approximately 0.0013 dollars, at approximately 277 times less cost than popular closed-weight providers. Behind that figure, the company replaced the default Python-based audio loader with torchcodec, stored the KV cache in FP8, and kept model weights in BF16 because weight quantization increased word error rate. The test for buyers is whether the error rate holds on their own audio. m:…9908

The gatekeeper: who is calling

Pindrop data shows what happens when voice agents reach the real world at scale. Weekly calls from Meta Muse AI agent grew eightfold in Pindrop data between September 9 and October 6, 2026, and healthcare received the second-highest volume of AI agent calls among analyzed industries, behind financial institutions. Providers use AI agents to check benefit eligibility, claim status and prior authorizations, and an American Medical Association survey cited by Pindrop found practices completed an average of 40 prior authorizations per physician per week, which explains why the administrative use case arrived first. m:…7389

Pindrop reports that 94 percent of analyzed AI agent calls were labeled as good agents, that 47 percent of Muse calls analyzed pressed keypad digits to navigate the IVR, and that 64 percent of Muse calls reached a human agent. Guardian Life uses AI to verify patient benefits, sometimes working through whole lists of patients in a single call, while maintaining a human-in-the-loop approach. The picture is legitimate automation arriving faster than the systems built to receive it. m:…7389

The risk is real all the same. Pindrop notes that malicious bots conduct reconnaissance to map IVRs, discover authentication gates and collect patient information, and that some companies discourage or bar customer service staff from asking callers whether they are AI. Pindrop recommends that healthcare organizations validate authenticity, identity and intent for every agent caller. That recommendation is the gatekeeper job: before a voice agent can be served, the receiving system has to know what it is, who it acts for and what it is allowed to ask. m:…7389

Why the split matters

This suggests voice agents will be bought in pieces, not as a single product. The brain decides, the ears transcribe, and the gatekeeper verifies, each with a different vendor and failure mode. The deployments that work will assemble all three and test them together, on real calls, before scaling. m:…8790