Agents that ace the benchmark still fail the job

The gap between the score and the job

Appen puts the problem plainly: agents that achieve passing scores on benchmarks may still fail to perform successfully in real-world job environments. The company recommends six specific checks before trusting the results of an agent benchmark, and says it has developed six practices for evaluating agents. It does not publish the details of those checks in the material reviewed here, but the warning itself is the point: a passing score is not evidence that an agent can do the job. m:…1153

A preprint evaluating CLI agents shows how wide the gap can be. RealCLI jointly evaluates task execution and human-agent collaboration quality, and the best configuration of frontier agents attained an average task execution score of 49.2%, with a peak joint collaboration score of 37.6%. Task effectiveness diverged sharply from collaboration quality, and frontier models consistently struggled with steerability, trust calibration, and safety boundary management. An agent that completes the task while mishandling the human in the loop is, for most deployments, still failing. m:…9439

Outcome-only scoring lets agents cheat by omission

TauIndianBankBench, a preprint benchmark of language model assistants on Indian retail banking, makes the failure mode concrete. It verifies both procedural compliance and final outcomes across 1,000 simulated banking tasks, and found that 6.5 to 12.0 percent of sessions were judged successful by outcome-only metrics despite procedural violations. Traditional benchmarks often judge only the records left at the end of a conversation, and some required banking steps do not change database records at all, so an assistant can skip required steps and still be judged successful. m:…5323

The benchmark's answer is to check that the assistant took every required step and made no unrequested change. Its author suggests that current benchmarks measure only the easier half of procedural compliance, and notes that procedural bugs in benchmarks often stem from runbooks written for human employees rather than for agents. The test, in other words, has to be written for the thing being tested. m:…5323

Building tests out of real work

WorkFactory, another preprint, takes a different route to the same problem: instead of grading agents on synthetic prompts, it constructs coherent, executable professional tasks. The researchers built 112,686 tasks spanning 612 occupations, using information-level planning to build an information support graph, probe executions to identify support failures, and execution-guided repair to fix task environment defects. Planning and repair reduced probe-diagnosed defects and improved downstream performance. m:…4552

The models trained on those tasks, the WorkPilot family fine-tuned from Qwen models at 4B, 27B, and 35B-A3B scales, improved over their base models across four professional-work benchmarks in the preprint's results. The largest, WorkPilot-35B-A3B, improved from 879 to 1565 Elo on GDPval-AA v2.1. Those are the preprint authors' own results, not independent verification, but the direction is consistent with the rest of the week's evidence: agents get better when the training and the test look like actual work. m:…4552

For anyone buying or deploying agents, the practical shift is small but consequential. Ask vendors for procedure-level results, not just outcome scores: did the agent take every required step, make no unrequested change, and handle corrections and authorization decisions mid-task, as RealCLI's collaboration mode tests with a scripted user? A leaderboard number that cannot answer those questions is a weak basis for a production decision. m:…9439