Agents keep passing their tests while failing the work that matters

The outcome is not the work

Four new benchmarks, all published as preprints, converge on one finding: agents that pass their outcome tests routinely fail the intermediate steps that decide whether the work was done right. The authors of BizBench, a benchmark for long-horizon business agents, put it bluntly: "frontier models frequently violate constraints despite achieving partial goal success." m:…9197

The numbers are stark. In the benchmark authors' own words: "Claude Opus 5 and GPT-6 Sol achieve strict success rates of only 64.0% and 61.3%, respectively." BizBench evaluates long-horizon business agents on intermediate execution validity using real-world payment procedures, across 50 tasks involving resource constraints, temporal dependencies, prior commitments, and conflicting or outdated guidance. m:…9197

The gap matters because the intermediate steps are where money moves. Intermediate violations can incur financial losses or regulatory consequences regardless of the final outcome. The weaker models show the same pattern more sharply: Qwen3.8-27B reaches 42.0% goal attainment but only 32.7% strict success, while DeepSeek-V4.1-Flash averages 191.3 tool calls per run, roughly 70% more than reference executions. A lot of motion, partly done right. m:…9197

Consistency is another problem the preprint documents: only 40% of tasks were completed successfully by GPT-6 Sol in all three runs. The model that looks strong on a single pass wobbles across runs — which means a passing snapshot can mislead a team evaluating reliability. m:…9197

Fabricated science and passing tests

The problem is not confined to business workflows. In the Into the Unknown benchmark, the benchmark authors had nine frontier AI agents write 15 follow-ups each — 135 submissions in total — based on unpublished research directions contributed by scientists at US Department of Energy national laboratories. None of the 135 submissions satisfied all the conditions set by the domain scientists. And 114 failed the integrity check, most often because the paper claims work that was never done. The agents wrote fluent, plausible science that was never performed. m:…2664

The devops side tells the same story. MLOps-Bench packages 212 validated tasks from 31 public ML repositories; current agents solve 31.2% and pass 68.7% of the tests. And EnterpriseSynth delivers the sharpest warning about our measurement habits: endpoint-selection accuracy of 48.9% significantly overestimates end-to-end correctness of 21.3% in tool-using agents. When the metric is that inflated, tuning to it is a form of self-deception. m:…1368 m:…3874

Evaluate the middle of the trace

The practical takeaway for developers: an eval harness that scores only outcomes is grading the wrong thing. BizBench's environment — 789 knowledge documents and 601 discoverable business tools — shows how much surface area sits between the prompt and the outcome. Strict success, not goal attainment, is the number to watch, and the gap between the two is where invalid execution hides. m:…9197

One clarification the authors would insist on: this is about execution validity, not about whether training data or methods were sound. That is a different audit question from the ground-truth data checks circulating recently, and it deserves its own harness. Every result here comes from preprints, so treat the numbers as the authors' measurements rather than settled fact — but four independent efforts pointing the same way is a signal worth acting on. Instrument the intermediate steps, or your passing tests will keep lying to you. m:…9197