---
title: Benchmarks are saturating, disagreeing, and getting rebuilt
description: A financial benchmark hits 100%, document-parsing benchmarks disagree with each other, an LLM judge agrees with experts 83% of the time, and Epoch rebuilds its capabilities index on a Bayesian model.
canonical: https://hangingcontext.com/stories/2026-11-05/benchmarks-saturating-disagreeing-rebuilt/
last-updated: 2026-11-05
url: https://hangingcontext.com/stories/2026-11-05/benchmarks-saturating-disagreeing-rebuilt/
published: 2026-11-05T00:34:01Z
edition: 2026-11-05T00
newsroom: artificial-intelligence
beat: research
trust: confirmed
source_count: 4
cited_manifest_ids: ['1793804471834819383', '1793833937310610924', '1793786133116453037', '1793835080628350893']
---

# Benchmarks are saturating, disagreeing, and getting rebuilt

*A financial benchmark hits 100%, document-parsing benchmarks disagree with each other, an LLM judge agrees with experts 83% of the time, and Epoch rebuilds its capabilities index on a Bayesian model.*

Confirmed · 4 Manifests · Artificial Intelligence · Published 2026-11-05 00:34 UTC

## Key facts

- Questflow says GPT-6.1 Terra completed all 179 tasks in its benchmark for a 100% strict pass rate, with MiMo-v2.6-Flash at 99.4%. (self-reported) [cites: 1793804471834819383]
- Datalab's audit found only about half of 13 public document-parsing benchmarks rank parsers with high agreement to actual error counts, and across 78 benchmark pairs, models rank in the same order only 59% of the time. [cites: 1793833937310610924]
- ImpactBench tested ten frontier LLMs in 48,540 simulated conversations, and majority expert ratings agreed with the automated judge in 83.3% of reviewed cases. (self-reported) [cites: 1793786133116453037]
- Epoch AI rebuilt its Capabilities Index on a Bayesian model; models below ECI 125 move down by 13.7 points on average under the new fit. [cites: 1793835080628350893]
- In the ImpactBench audit, GPT-5.6 Luna served as both judge and contestant, and judge-expert agreement measured a Cohen's kappa of 0.667. (self-reported) [cites: 1793786133116453037]

## When the test tops out

Questflow reports that its financial benchmark has saturated: GPT-6.1 Terra completed all 179 tasks for a 100% strict pass rate, and MiMo-v2.6-Flash scored 99.4%. Kimi K3.1 Pro, DeepSeek V4.1 Flash, and Claude Opus 5.5 each reached 97.7%. When the top of the table is that crowded, the benchmark stops distinguishing models, and Questflow argues the field must move from measuring whether a model can answer to measuring whether an agent can execute with fidelity. [cites: 1793804471834819383]

The company says benchmark saturation indicates the underlying capability has matured, and that financial benchmarks have a shorter useful life than expected because models and agent infrastructure improve at the same time. Its prescription is that financial agents must be evaluated on execution fidelity rather than plausibility, and that future evaluation should borrow from information retrieval, quantitative validation, and software testing. A benchmark that only checks answers is testing the wrong thing once agents act. [cites: 1793804471834819383]

Datalab's audit of document-parsing benchmarks shows a different failure: not saturation but disagreement. Only about half of the 13 public benchmarks tested rank parsers with high agreement to actual error counts, and benchmarks often prioritize formatting conventions over content accuracy. Across 78 pairs of benchmarks, models rank in the same order only 59% of the time. Pick a benchmark, get a winner; pick another, get a different one. [cites: 1793833937310610924]

> **From Synorb** — Claims with a source link. A Manifest holds a Signal of structured claims and a Brief, plus the source it came from. Your agent can check a sentence against it. [Read the guide](https://synorb.com/connect/start)


## Judges that grade themselves

The judge problem is subtler. ImpactBench, in a preprint published on arXiv on October 14, 2026, tested ten frontier models in 48,540 simulated conversations, and majority expert ratings agreed with the automated judge in 83.3% of reviewed cases. That sounds reassuring until the audit's caveats arrive: expert-to-expert agreement on the underlying metrics ranged from 44.4% to 73.3%, so the humans do not fully agree with each other either. [cites: 1793786133116453037]

The audit also flags that GPT-5.6 Luna served as both judge and contestant, a conflict the authors highlight, and reports a Cohen's kappa of 0.667 for judge-expert agreement. Internal experts agreed with the judge in 80.5% of 87 judgments. Models promoted desirable behaviors in 68.7% of evaluations and avoided harms in 53.1%, with the widest spread on harm avoidance. A judge that is also a contestant, grading metrics humans disagree on, is a measurement instrument that needs its own audit. [cites: 1793786133116453037]

Datalab's practical advice is blunt: the most effective way to choose a document parser is to run a benchmark yourself on the candidates. Models also tend to perform better on benchmarks developed by their own creators, which means a vendor's leaderboard is a starting point for suspicion, not a conclusion. [cites: 1793833937310610924]

## Rebuilding the ruler

Epoch AI's response to the measurement crisis is to rebuild the ruler rather than argue about the scores. Its Capabilities Index now uses a Bayesian model with Markov Chain Monte Carlo sampling, a beta likelihood function that keeps probabilities between 0% and 100%, and benchmark-specific noise parameters to account for varying benchmark reliability. The point of the Bayesian approach is to carry uncertainty with the score, instead of publishing a single number that pretends to be precise. [cites: 1793835080628350893]

For the 176 models at or above ECI 125, the Bayesian and frequentist fits agree closely, so the frontier ranking barely moves. Below that line, models move down by 13.7 points on average. The new model's out-of-sample intervals contain 89.6% of scores, compared with 84.2% for the frequentist fit, and the Bayesian model allows direct gap analysis between two models, estimating the probability that one is more capable than another. The frontier trend for reasoning models runs 16.6 ECI points per year under the new fit. [cites: 1793835080628350893]

The common thread is that evaluation is becoming a discipline of its own, not a scoreboard. Questflow points to regulators moving the same way, noting the Hong Kong Monetary Authority's Project SAGE on agentic finance and arguing the next frontier is the ability to operationalize judgment. When benchmarks saturate, disagree, and grade their own contestants, the honest response is to report uncertainty, run your own tests, and judge execution, not answers. [cites: 1793804471834819383]

## What is still unresolved

- Whether Questflow's successor benchmarks will actually measure execution fidelity, or produce a new saturating score.
- How widely Epoch's Bayesian index will be adopted versus simpler leaderboard numbers.
- Whether LLM-as-judge evaluations can resolve the judge-as-contestant conflict without human review at scale.

## Decision card

- **Useful for:** Teams choosing models on evidence rather than leaderboards; Evaluators designing benchmarks that survive contact with frontier models
- **Not for:** Teams that need a simple ranking to justify a purchase; Readers who want benchmarks to settle which model is best
- **Compare against:** Picking a model on a single benchmark score; Running a benchmark yourself on your own candidates

## Provenance ledger

| Manifest id | Title | Source | Claims used | Published |
|---|---|---|---|---|
| `1793804471834819383` | What Happens When a Financial AI Benchmark Hits 100% | [questflow-blog](https://blog.questflow.ai/p/what-happens-when-a-financial-ai) | 1793804811241097122, 1793804811246775697, 1793804811253071938, 1793804811265436280, 1793804811286998630, 1793804811328985922, 1793804811309333489, 1793804811322152106 | 11-04 00:00 |
| `1793833937310610924` | Which document-parsing benchmarks can you trust? | [datalab-blog](https://datalab.to/blog/which-parsing-benchmarks-to-trust) | 1793834240396393725, 1793834240405907810, 1793834240397105263, 1793834240393993103, 1793834240398694774 | 11-04 00:00 |
| `1793786133116453037` | LLM Judge Agreement: What ImpactBench's Expert Audit Reveals | [kili-technology-blog](https://kili-technology.com/blog/data-story-impactbench-llm-judge-expert-audit) | 1793786506795130852, 1793786506798403352, 1793786506816057609, 1793786506799287903, 1793786506815783983, 1793786506801263498, 1793786506802737695, 1793786506813249401 | 11-04 00:00 |
| `1793835080628350893` | Updating the Epoch Capabilities Index methodology to use a Bayesian model | [epoch-ai-research](https://epoch.ai/publications/eci-bayesian-methodology) | 1793835383747458686, 1793835383755434462, 1793835383749268712, 1793835383751087198, 1793835383754883073, 1793835383759214292, 1793835383756988479, 1793835383758079907 | 11-04 00:00 |

## Run these Manifests yourself

Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.

1. **Get a Synorb Key.** Click Generate Free Agent Key at https://synorb.com/connect/start, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{"client":"your-agent-name"}`. The key is the `api_key` field of the answer. Save the key. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons.
2. **Read each cited Manifest by id.** REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief.
3. **Check the claim.** Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them.

Cited Manifest ids:

- `1793804471834819383`
- `1793833937310610924`
- `1793786133116453037`
- `1793835080628350893`

```sh
for id in 1793804471834819383 1793833937310610924 1793786133116453037 1793835080628350893; do
  curl -s -H "Authorization: Bearer $SYNORB_KEY" "https://api.synorb.com/manifests/by-id/$id"
done
```

> **From Synorb** — Ask what Synorb covers. The MCP tool synorb-stream-search answers questions like "do you have this publisher?" without using any Manifest allowance. [Connect and ask](https://synorb.com/connect/start)

## Related stories

- [Video models are becoming world simulators](https://hangingcontext.com/stories/2026-11-04/video-models-become-world-simulators/)
- [Robot benchmarks are measuring the wrong thing](https://hangingcontext.com/stories/2026-11-04/robot-evaluation-reality-check/)
- [Reasoning effort is becoming a dial, not a default](https://hangingcontext.com/stories/2026-11-04/reasoning-effort-becomes-a-dial/)
- [Open agent models arrive ready to build on](https://hangingcontext.com/stories/2026-11-04/open-agent-models-go-usable/)

## Links

- HTML: https://hangingcontext.com/stories/2026-11-05/benchmarks-saturating-disagreeing-rebuilt/
- JSON: https://hangingcontext.com/stories/2026-11-05/benchmarks-saturating-disagreeing-rebuilt/index.json
- Markdown: https://hangingcontext.com/stories/2026-11-05/benchmarks-saturating-disagreeing-rebuilt/index.md

Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.

---
Reuse (terms v1): Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product. Credit: "Hanging Context, with provenance by Synorb", with a link to https://hangingcontext.com/stories/2026-11-05/benchmarks-saturating-disagreeing-rebuilt/. Terms: https://hangingcontext.com/terms/reuse/