The benchmark says pass. The real world says otherwise.

The score is not the skill

A new preprint treats benchmark contamination as a property of five things at once: the model, the evaluation harness, the elicitation budget, the sampled population and the contamination status. A score, in that framing, means little on its own: contamination is a joint property of all five, not a flaw in any one of them. m:…9085

The reporting is in poor shape: only 13% of surveyed research documents report elicitation budgets, and inter-coder reliability for assessing contamination reporting is low, with a median kappa of 0.21. m:…9085

One run is not enough

A second preprint argues a Monte Carlo protocol with 100 repeated runs is required to accurately assess LLM reliability in data science tasks. The point is that a single run is not a measurement: responses for data visualisation and supervised learning tasks varied substantially at medium and hard difficulty, so a model can pass once and fail the next time. m:…8168

The same problem shows up for agents. VibeLifeBench tests 200 long-horizon tasks in a simulated world of 22 mock services that advance on their own clock. Existing agent evaluations rely on short, self-contained requests in static environments, and seven frontier models tested on VibeLifeBench performed poorly on proactivity and persistence. m:…8183

Static success, dynamic failure

In protein design, the gap is measured precisely. BILBY applies molecular dynamics to designs that passed the existing MotifBench benchmark: of 55 statically solved design-problem pairs, 31 satisfied the molecular-dynamics criterion and only 8 passed all checks. The authors put it plainly: "Static refolding success therefore provides useful evidence of local motif recovery but does not establish dynamical fidelity of the full scaffold." RFDiffusion and Proteina solved more targets, while ESM3 produced more unique successful scaffolds on the targets it solved. BILBY also introduces BILBY-PRO, a benchmark of 220 structurally conserved functional motifs derived from PROSITE-linked experimental structures. m:…2293

In finance, the benchmark FinDendro tests whether models know the limits of their own knowledge, using 2,226 dated US facts from 1999 to 2026. GPT-5 answered 76% of the controlled facts correctly while claiming to know only 38% — and answered 61% of the facts it said it did not know. Knowledge claims persisted up to seven quarters beyond the measured knowledge boundary, and telling the model to answer as of a historical date removed most claims of prior knowledge while leaving accuracy unchanged. Backtests run inside a model's training window, the authors conclude, cannot be certified leak-free by self-report, instruction or stated cutoff. m:…2287

What the fix looks like

Artificial Analysis, which runs its own model leaderboard, released version 2 of its Decision Index. The update adds a Decision Score that rewards correctness and calibrated confidence, and raises private datasets to 45% of the index, including a new private benchmark of 1,500 documents. The update also adds AA-Symmetry, which tests consistency across 120 questions with 440 variations. m:…7850

The results show how much the choice of test changes the answer. Drex Max v1.1 and GLiDE v1.2 both scored 73, behind Claude Haiku 5.5 (Max) at 77 — but cost per million tasks varied by more than 60 times across the models, from $102 for Drex Max to $1,530 for GPT-6 Luna. Mammoth 27B scored 72 on the Index and ran 118 times faster than GPT-6 Luna (Max), and Celeris-1.1 Decision recorded the lowest time per task at 56 milliseconds. The ranking depends on what you measure, and the cheapest model that clears your bar may be the one the old leaderboard hid. m:…7850