The score says the model works. The work says otherwise.

The shortcut problem

Delete every noun from a benchmark question and a good language model still gets most of it right. In a preprint posted this week, researchers tested eight large language models on standard language-understanding benchmarks with the input corrupted — nouns removed, words shuffled — and found the models retained 36 to 72 percent of their performance on MNLI, a widely used benchmark. On ANLI, a harder benchmark built to resist shortcuts, they kept only 14 to 29 percent. The gap is the finding: the models are not reasoning through the corrupted questions, they are leaning on dataset-specific cues that survive the damage. m:…1942

The same preprint rules out the usual excuse. The researchers found no link between pre-training contamination, measured by counting overlapping n-grams, and how well a model held up under corruption. A model that had seen the test data was no more robust than one that had not. Robustness to this kind of damage, they conclude, is a property of the benchmark's design, not of the model's intelligence — except in models that explicitly reason before answering. The score, in other words, measures the test as much as the model. m:…1942

The judge problem

The problem compounds when the judge is itself a model. A second preprint, testing evaluators on tau2-bench, an agentic benchmark, found that LLM judges agreed with the benchmark's own scoring on only 56.2 percent of episodes. The disagreement is not academic: the judges proved unreliable at detecting forbidden database writes, the kind of mistake that matters in production, while rule-based boundary checks caught them. And the usual safety gates — verification, correction, safe termination — did not significantly reduce the rate of forbidden writes, the authors report, often because the evidence digest handed to the gate had lost the critical context. m:…9313

Two other papers this week try to fix the judge rather than replace it. One, AGORA, learns how reliable each judge is from the pattern of its answers, using item-response theory, without needing human labels — and beats the standard Dawid-Skene aggregation on small evaluation sets of a few hundred items or fewer. The other, on decision models in long-horizon agents, finds that judging whether each step of a trajectory helped is a better signal than predicting whether the task succeeded, and that a model too weak to act well can still teach a policy: its JEV method lifted a small Qwen2.5-1.5B policy from 72.8 to 82.9 percent on ALFWorld and from 56.8 to 71.4 percent on WebShop. m:…0107 m:…2584

The professional standard

The sharpest illustration of the gap between scores and work comes from Centific, whose researchers presented work at NeurIPS 2026. Their strongest model scored 80 percent accuracy on final answers for healthcare tasks — and passed the full professional standard only 18 percent of the time. The deliverable, not the answer, is where models fail. An on-device assistant evaluated by checking the actual device state passed 10 of 20 tasks, and passed 9 of 14 action tasks but only 1 of 6 restraint tasks — doing things, in other words, is easier for it than knowing when not to. The best voice agent tested completed 49 percent of tasks on the first attempt across 3,825 conversations. m:…3844

What replaces the score

The week's most concrete alternative comes from LMArena, whose researchers propose judging computer-use agents by real interaction rather than by code or screenshots. Their Interaction-as-a-Judge method raised broken-requirement detection from 63.0 to 71.1 percent compared with a code-only judge, and lifted requirements fixed per repair iteration from 55.4 to 75.5 percent. Reinforcement learning with an interaction-only reward improved a Qwen-3.5-9B model's win rate by more than 50 Arena leaderboard points, and a 9-billion-parameter reward model trained this way reached 70.8 percent best performance, ahead of frontier models, the team reports. The pattern across the week is consistent: the score that is easiest to compute is the least trustworthy, and the evaluations that hold up are the ones that watch the work happen. m:…8575