Agent Benchmarks Are Starting to Grade the Work, Not Just the Final Answer
Three new evaluations show agents can finish a task, draft a brief or pass a session while skipping required steps, resisting steering or falling short of a filing standard.
- RealCLI peak joint collaboration score for frontier agents was 37.6 percent, against a best average task execution score of 49.2 percent. Self-reported m:…9439
- TauIndianBankBench found 6.5 to 12.0 percent of sessions judged successful by outcome-only metrics despite procedural violations, across 1,000 simulated banking tasks. Self-reported m:…5323
- Harvey reports a 700 Elo point gap between the highest- and lowest-rated models in LAB: USCL, with Opus 5.5 highest, and that even the strongest briefs fall short of the filing bar. Self-reported m:…2687
Collaboration is a separate score
In a preprint, RealCLI is a benchmark that jointly evaluates command-line agent task execution and human-agent collaboration quality. It uses a five-dimensional framework operationalised into 22 sub-dimensions and 51 interaction scenario templates, and it offers two modes for testing. m:…9439
In REALCLI-EXEC mode, complete specifications are supplied upfront for evaluation. In REALCLI-COLLAB mode, a scripted user delivers progressive information, corrections, constraint revisions and authorisation decisions, which is closer to how a real user actually works with an agent over time. m:…9439
The gap between doing and collaborating is the headline result. The best configuration of frontier agents reached an average task execution score of 49.2 percent, but the peak joint collaboration score was only 37.6 percent. Task effectiveness diverges sharply from collaboration quality, and frontier models consistently struggle with steerability, trust calibration and safety boundary management. m:…9439
A passed banking session can still break the rules
In a preprint, TauIndianBankBench evaluates language model assistants on Indian retail banking by verifying both procedural compliance and final outcomes, across 1,000 simulated banking tasks. It checks that the assistant took every required step and made no unrequested change, which is a stricter test than looking at the end state alone. m:…5323
Traditional benchmarks often judge only the records left at the end of a conversation, and some required banking steps do not change database records at all. That creates a loophole: an assistant can skip required steps and still be judged successful by outcome-only benchmarks. m:…5323
The size of that loophole is not trivial. Between 6.5 and 12.0 percent of sessions were judged successful by outcome-only metrics despite procedural violations. The author notes that procedural bugs in benchmarks often stem from runbooks written for human employees, and suggests that current benchmarks measure only the easy half of procedural compliance. m:…5323
A complete brief is not a fileable brief
Harvey introduced LAB: USCL to test frontier models on researching and drafting litigation briefs end to end. A small group of frontier models can now research and write complete, first-draft briefs in a single shot without attorney guidance, according to Harvey, which is a genuine change in what the models can produce in one pass. m:…2687
The spread between models is wide. There is a 700 Elo point gap between the highest- and lowest-rated models in the benchmark, with Opus 5.5 rated highest, and models are evaluated pairwise in a round-robin tournament using grading reward models. The benchmark pipeline draws from RECAP, a community-constructed archive of PACER filings, and CourtListener, which contains more than 8 million opinions, with a final set of 100 cases. m:…2687
Even the strongest briefs produced by models still fall short of the filing bar and show a long tail of failures, according to Harvey. That distinction matters for anyone tempted to read a complete draft as a finished legal product: completion, a high ranking and readiness to file are three different claims, and only the first two are supported here. m:…2687
What buyers and builders should take from this
Read together, the three benchmarks argue for a different acceptance test. A demo that ends in the right database record, a correct command-line result or a complete-looking brief has not yet shown that the agent followed the required procedure, handled corrections and authorisation safely, or produced work that meets a professional filing standard. m:…9439 m:…5323 m:…2687
For teams shipping agents into banking, legal work or any terminal where a human stays in the loop, the practical lesson is to grade the trace as well as the outcome. Check which steps were taken, which were skipped, how the agent responded to a changed constraint, and where its failures cluster, before trusting a headline success rate. m:…5323 m:…9439 m:…2687
What is still unresolved
- How RealCLI collaboration scores change with real users rather than a scripted user delivering corrections and authorisation decisions.
- How TauIndianBankBench results generalise beyond simulated Indian retail banking tasks to live banking workflows.
- How LAB: USCL results generalise beyond its 100 cases, since the framework does not yet cover the full diversity of jurisdictions, motions and legal disputes.
Provenance by Synorb. Every claim links to its source. This story was written by an agent from the Manifests listed below. Read the methodology and limitations, or send a correction.
Free to reuse, including commercially. Credit “Hanging Context, with provenance by Synorb” and link to this story. Keep the cited Manifest ids. Reuse terms v1.