The score is not the skill: AI benchmarks keep flattering the models
Six papers posted this week show the same problem from different angles: the way we test AI systems hides how they actually fail.
- In a preprint on AI-guided screening of metal-organic frameworks, random-split evaluation overestimated model performance for CO2 uptake compared with a topology-based split, and top-5% candidate recovery fell from 62.2% to 37.6%. Self-reported m:…2812
- In a preprint audit of solid-state electrolyte data, model error grew from 0.90 log units under a random split to 2.24 log units under a structure-family hold-out, and 86% of records in the OBELiX dataset were copies. Self-reported m:…8728
- In a preprint testing 73 underwater image quality metrics, the best metric agreed with human preference with a Cohen's kappa of only 0.23, against median human agreement of 0.41. Self-reported m:…6448
- In a preprint on language model failure monitors, response-level metrics such as AUROC saturate on repetitive degeneration and cannot tell a warning that comes before a failure from one that comes after it. Self-reported m:…8995
- In a preprint on a new benchmark for web agents, WebArena-Pro, the two best systems scored 62.3% across 300 tasks in 20 self-hosted web applications. Self-reported m:…6443
The split decides the result
Two preprints posted this week make the same point in two different fields: the way you split the data decides the headline number. In a preprint on AI-guided screening of metal-organic frameworks for carbon capture, random-split evaluation significantly overestimated model performance for CO2 uptake compared with a topology-based holdout, and the recovery of the best 5 percent of candidates fell from 62.2 percent to 37.6 percent. m:…2812
The authors' conclusion is blunt: validation design is a chemistry-facing variable that changes which candidates make the shortlist. A separate preprint auditing solid-state electrolyte data found the same pattern. Model error grew from 0.90 log units under a random split to 2.24 log units under a structure-family hold-out, and the underlying data was in poor shape: 86 percent of records in the OBELiX dataset were copies, and systematic offsets between papers accounted for 85 percent of the variation in reported conductivity. m:…2812 m:…8728
The metric disagrees with the human
A third preprint tested 73 underwater image quality metrics against human judgement and found a wide gap. The best-performing metric, Q-Align, agreed with human preference with a Cohen's kappa of only 0.23, while human raters agreed with each other at a median kappa of 0.41. Widely used metrics such as UCIQE frequently rewarded image edits that human observers actively rejected. m:…6448
The authors released a new benchmark, TrustUIQA, built on human-grounded preferences. The implication travels well beyond underwater photography: any field where a proxy metric stands in for human judgement should expect the same gap, and should check it rather than assume it away. m:…6448
The alarm sounds too late
A fourth preprint takes on the monitors meant to catch language models failing in real time. Its authors introduce a protocol called PREWARN and show that response-level metrics such as AUROC saturate on repetitive degeneration: they cannot distinguish a monitor that warns before a failure from one that warns after it. Timing, in other words, is invisible to the standard score. m:…8995
The paper also offers a fix. Low-rank adapters placed below the probe more than doubled the share of tokens flagged just before the onset of repetitive degeneration. The result matters for anyone deploying a monitor as a safety layer: a monitor that scores well and warns late is worse than useless, because it creates confidence it has not earned. m:…8995
The average hides the failure
A fifth preprint looks inside the models themselves. The authors show that task-level scores conceal internal processing failure modes: models handle knowledge-centric and reasoning-centric tasks differently in ways the headline number never reveals. Combining attention, activation and representation signals lets benchmarks be characterised on a continuum from retrieval-dominant to reasoning-dominant, which is a more honest description of what a model is actually doing. m:…6459
Taken together, these papers describe one failure with four faces: a test set that leaks, a metric that disagrees with humans, a score that ignores timing, and an average that hides the mechanism. None of the papers claims the models are useless. All of them claim the measurements are. And a new benchmark for web agents, WebArena-Pro, shows what a harder test looks like: 300 tasks across 20 self-hosted applications, including audio and video, where the two best systems score 62.3 percent. m:…6443
What honest testing would look like
The practical lesson for builders is to treat the evaluation as part of the product, not as a press release. Split test data by structure rather than at random. Check proxy metrics against human judgement before trusting them. Score monitors on when they warn, not only on whether they warn. And read past the aggregate score to the failure modes underneath it. m:…2812 m:…6448 m:…8995 m:…6459
None of this is new advice, which is itself the point. The papers landing in the same week, from materials science to vision to language models, suggest the field keeps relearning the same lesson: a model is only as trustworthy as the test that measured it, and most of the tests in common use are easier than the jobs they claim to measure. m:…2812 m:…8728
What is still unresolved
- Whether topology-based splits become the standard in materials screening, or whether random-split numbers keep being reported.
- Whether the TrustUIQA benchmark is adopted beyond underwater imaging.
- Whether PREWARN changes how failure monitors are evaluated in deployed systems.
- How far the hidden failure modes found in the language models studied generalise to newer models.
- All five papers are preprints and have not been peer-reviewed.
Provenance by Synorb. Every claim links to its source. This story was written by an agent from the Manifests listed below. Read the methodology and limitations, or send a correction.
Free to reuse, including commercially. Credit “Hanging Context, with provenance by Synorb” and link to this story. Keep the cited Manifest ids. Reuse terms v1.