Benchmarks are saturating, disagreeing, and getting rebuilt

When the test tops out

Questflow reports that its financial benchmark has saturated: GPT-6.1 Terra completed all 179 tasks for a 100% strict pass rate, and MiMo-v2.6-Flash scored 99.4%. Kimi K3.1 Pro, DeepSeek V4.1 Flash, and Claude Opus 5.5 each reached 97.7%. When the top of the table is that crowded, the benchmark stops distinguishing models, and Questflow argues the field must move from measuring whether a model can answer to measuring whether an agent can execute with fidelity. m:…9383

The company says benchmark saturation indicates the underlying capability has matured, and that financial benchmarks have a shorter useful life than expected because models and agent infrastructure improve at the same time. Its prescription is that financial agents must be evaluated on execution fidelity rather than plausibility, and that future evaluation should borrow from information retrieval, quantitative validation, and software testing. A benchmark that only checks answers is testing the wrong thing once agents act. m:…9383

Datalab's audit of document-parsing benchmarks shows a different failure: not saturation but disagreement. Only about half of the 13 public benchmarks tested rank parsers with high agreement to actual error counts, and benchmarks often prioritize formatting conventions over content accuracy. Across 78 pairs of benchmarks, models rank in the same order only 59% of the time. Pick a benchmark, get a winner; pick another, get a different one. m:…0924

Judges that grade themselves

The judge problem is subtler. ImpactBench, in a preprint published on arXiv on October 14, 2026, tested ten frontier models in 48,540 simulated conversations, and majority expert ratings agreed with the automated judge in 83.3% of reviewed cases. That sounds reassuring until the audit's caveats arrive: expert-to-expert agreement on the underlying metrics ranged from 44.4% to 73.3%, so the humans do not fully agree with each other either. m:…3037

The audit also flags that GPT-5.6 Luna served as both judge and contestant, a conflict the authors highlight, and reports a Cohen's kappa of 0.667 for judge-expert agreement. Internal experts agreed with the judge in 80.5% of 87 judgments. Models promoted desirable behaviors in 68.7% of evaluations and avoided harms in 53.1%, with the widest spread on harm avoidance. A judge that is also a contestant, grading metrics humans disagree on, is a measurement instrument that needs its own audit. m:…3037

Datalab's practical advice is blunt: the most effective way to choose a document parser is to run a benchmark yourself on the candidates. Models also tend to perform better on benchmarks developed by their own creators, which means a vendor's leaderboard is a starting point for suspicion, not a conclusion. m:…0924

Rebuilding the ruler

Epoch AI's response to the measurement crisis is to rebuild the ruler rather than argue about the scores. Its Capabilities Index now uses a Bayesian model with Markov Chain Monte Carlo sampling, a beta likelihood function that keeps probabilities between 0% and 100%, and benchmark-specific noise parameters to account for varying benchmark reliability. The point of the Bayesian approach is to carry uncertainty with the score, instead of publishing a single number that pretends to be precise. m:…0893

For the 176 models at or above ECI 125, the Bayesian and frequentist fits agree closely, so the frontier ranking barely moves. Below that line, models move down by 13.7 points on average. The new model's out-of-sample intervals contain 89.6% of scores, compared with 84.2% for the frequentist fit, and the Bayesian model allows direct gap analysis between two models, estimating the probability that one is more capable than another. The frontier trend for reasoning models runs 16.6 ECI points per year under the new fit. m:…0893

The common thread is that evaluation is becoming a discipline of its own, not a scoreboard. Questflow points to regulators moving the same way, noting the Hong Kong Monetary Authority's Project SAGE on agentic finance and arguing the next frontier is the ability to operationalize judgment. When benchmarks saturate, disagree, and grade their own contestants, the honest response is to report uncertainty, run your own tests, and judge execution, not answers. m:…9383