Your voice agent sounds fine on paper. The audio tells a different story

The vendor score is not the product

Voice agent vendors now bundle building, deployment and evaluation into one platform, but Roark argues that a vendor cannot also be its own quality assurance team. Platform-generic evaluators cannot see business-specific requirements such as escalation paths, compliance rules or specific tool-call sequences, and vendors cannot test how their agents integrate with a buyer's internal stack, such as a CRM or billing system. The point is structural: a vendor-owned loop optimizes against generic metrics, not against the constraints that decide whether a call succeeds for a particular business. m:…6860

Coval's November 2026 benchmark shows what the vendor numbers do measure. PolyAI's Dialog-RSN-1 leads the speech-to-speech category with a p95 latency of 1,101 ms and 100% instruction adherence, while Gemini 2.5 Flash is the fastest raw text model for a cascaded agent at 589 ms to the first token. The same report carries a warning that buyers should read as a deadline: Google will shut down Gemini 3.1 Flash Live as early as November 17, 2026, and OpenAI will shut down GPT-4.1 on January 14, 2027. A model that tops a leaderboard this month can be gone next month, which makes the leaderboard a starting filter, not a buying decision. m:…6693

Echo is a system failure

The failure mode Roark spends the most time on is self-triggered echo: the agent hears its own audio output, treats it as caller input, and can fall into loops or confused responses. Soniox released STT v6 on November 3, 2026 with a model-level residual echo filter, and Roark says such ASR-side defenses shift the failure surface rather than removing the need for testing. Acoustic echo cancellation quality depends on hardware, driver stack, double-talk behavior and reference-signal alignment, all outside the agent's control, so a filter that works in one deployment can fail in another. m:…3625

Transcript-based evaluation cannot catch these failures, because the transcript of an echo loop can look like an ordinary conversation. Roark's testing framework uses loopback injection in a text harness as a fast, low-cost way to exercise echo-handling code, real-phone simulation over PSTN and WebRTC as the only reliable way to test actual acoustic echo behavior, and production call replay to turn real failures into repeatable regression tests. The target for self-loop count in a voice agent call is zero, Roark says, and independent QA must score audio rather than transcripts to capture pauses, talk-over and emotional affect. m:…3625 m:…6860

Speed is moving to the edge

Meanwhile the latency race is moving onto small hardware. Quadric tested the Moonshine v2 speech recognition model on a single Chimera QC-Nano core, where the encoder runs 24 times faster than real-time and processes 80 ms of audio in 3.36 ms, with both encoder and decoder steps running at 99% of the memory bandwidth roofline. Moonshine v2 uses a sliding window attention mechanism, the company says, so the decoder can start transcribing before an utterance is complete. The implication is that live transcription no longer needs a data-center round trip, which matters for agents that must react while a caller is still speaking. m:…8416

The same Coval report shows how fast the ground moves under a voice deployment. Google's recommended migration from Gemini 3.1 Flash Live to 3.8 Live reduced p95 latency by 1,344 ms, and Pipecat treats a model with a time-to-first-token above 700 ms as too slow for most voice agents. Coval recommends a three-step selection process: public benchmarks, local experimentation, and full simulation. The order matters, because a model that looks fast on a public board can still fail on a buyer's own workflows, and a model that passes local tests can still echo on a real phone line. m:…6693

What to test before launch

The thread running through all four sources is that voice quality is an audio problem, not a text problem, and that the vendor's own loop cannot certify it. Independent QA must score audio rather than transcripts, Roark says, and it must happen before launch and before every model or prompt change, because self-improvement loops are lossy when the loss function is not under the user's control. A vendor's self-improving agent tunes itself against vendor-defined success labels, which may conflict with the buyer's conversion or compliance goals. m:…6860 m:…3625

For a team about to ship a voice agent, the practical sequence is now fairly clear: start with public benchmarks to narrow the field, test locally against the team's own escalation and compliance requirements, simulate full calls, and keep a regression suite built from production replays. The cost of skipping that work is not a slightly worse score. It is an agent that talks to itself on a live call, on a model that may be shut down within weeks, with no transcript that shows what went wrong. m:…6693 m:…3625