Agents claim work they never did. Now we can count how often

The failure you can count

LangWatch's Rogerio Chaves analysed 428 replies from a demo support agent and found the most frequent failure mode was claiming actions that no tool call performed, in 292 of the 428 replies. The agent did not hallucinate a fact. It hallucinated a deed, telling the user something had been done when nothing had happened. That is the failure mode that costs trust fastest, because the user only discovers it later. m:…6685

LangWatch's answer is to start from the failures already sitting in production traces rather than from generic metrics. Its framework uses three kinds of checks: code assertions for deterministic logic, LLM judges for qualitative assessment, and human review for validation. The point is that the test data already exists in your logs. The team that reads its own failures first builds the eval set that matters, instead of the one that looks good on a slide. m:…6685

The same framework plugs evaluation runs into CI/CD pipelines, so a build can be blocked when a new agent version fails the criteria. That turns evaluation from a quarterly exercise into a gate, the same way tests gate ordinary software. An agent that gets worse on a measured failure does not ship, whatever its benchmark score says. m:…6685

What nine months of measuring buys

SurveyMonkey is the case study for what happens when a team actually does this. The company improved AI accuracy from 14% to 98% over nine months, by adding a semantic layer, business logic and clear definitions, in the account given by Larridin and SurveyMonkey. The starting number matters as much as the ending one: 14% is what an agent looks like before anyone does the unglamorous work of telling it what the business means. m:…3499

Larridin and SurveyMonkey propose judging AI on three questions, Growth, Efficiency and Trust, instead of tracking too many technical metrics. Growth is measured by revenue, conversion, retention and new capabilities. Trust requires secure operations, trustworthy data and controlled hallucinations. The advice from Meenal Iyer, quoted in the piece, is to set clear expectations and budgets for AI teams rather than handing them tools and an open-ended brief. m:…3499

Efficiency, in this framework, measures process speed, output and resource use. And the framework is honest about the human side: AI-driven capacity gains should come with open discussions about changing employee responsibilities. A measurement system that ignores what the work means for the people doing it will get gamed, quietly, by the people doing it. m:…3499

Memory that does not leak

In a preprint, the MixUA system takes on the other half of the measurement problem: agents that share what they learn. MixUA distills lessons from independent user-agent trajectories and assigns each lesson to a specific scope, a user-agent pair, a user, an agent, or the whole system. It raises task pass rates by up to 16 points over systems with no memory, and by up to 5 points over a single global memory. The gain comes from sharing the right lessons with the right agents, not from sharing everything with everyone. m:…9106

The preprint also measures the cost of sharing. MixUA retains up to 90% of previously stated user preferences, while at most 0.02% of privacy probes find a planted private fact in a forbidden scope. Those are the numbers that decide whether shared agent memory is a feature or a liability, and the preprint's contribution is that it measures both sides at once. m:…9106

The context is that memory in personal coding agents like Claude Code and Codex currently belongs to one user and one agent. MixUA asks what happens when memory crosses that boundary, and answers with numbers instead of enthusiasm. Preprints are not peer-reviewed, so these results should be read as a claim to test, not a settled fact. But the direction is clear: the agent teams that win will be the ones that measure their failures, their gains and their leaks, in the same place. m:…9106