The yardstick is failing its own test

Rubrics lose to comparisons

JudgmentBench, a preprint comparing rubric-based scoring with pairwise preference evaluation for quality assessment, uses expert legal annotations to test which method actually identifies quality differences in language model outputs. The finding is clear: comparative judgments recover intended quality ordering substantially better than rubrics under both a per-task rank-correlation metric and a per-judgment pairwise win-rate metric. And pairwise preference evaluation requires less than half the annotation time of rubric-based scoring. m:…2832

The result holds for both human annotators and LLM autograders, according to the authors. That matters because the usual defence of rubrics is that they are more systematic and more automatable. If pairwise comparison is both more accurate and cheaper, and works for machine judges too, the case for scoring outputs against a checklist gets thin. For teams building evaluation pipelines, the practical implication is to stop writing longer rubrics and start running more comparisons. m:…2832

Accuracy is the wrong number

The second preprint attacks a different assumption: that a good monitor is an accurate monitor. Researchers evaluating AI agent restart monitors argue that classification accuracy, and true-positive and false-positive rates alone, do not determine whether restarting an agent improves success. They evaluated 194 model executions on 19 selected SymPy issues from SWE-bench Verified, and the headline result is stark: a monitor with fixed 90% accuracy can swing from a five-percentage-point loss to a 35-point gain in success rate depending on the replacement attempt's residual-budget success probability. m:…1316

The replication problem makes it worse. A retrospective audit of 130 traces showed a 2.29-point apparent restart gain on one model and a 42.29-point loss on a second model, and initial restart gains observed in the study did not replicate on prospective task cohorts. Zero of 103 live comparisons in the study met the specified certification bar for restart or continue decisions. The authors propose a finite-sample, model-free decision audit that can certify a preference for restart or continue without additional model calls. The message is that teams should certify decisions, not monitors. m:…1316

Cheaper benchmarks, cheaper routing

The third preprint, BestRand, tackles benchmark compression — evaluating models on small benchmark subsets to save time and cost. Random sampling for benchmark compression results in a long-tailed error distribution, and instability in random sampling is inherent to modern foundation models, according to the authors. BestRand preserves random sampling efficiency while reducing high-error tails: it reduces high-error risk by 50.2% compared with competitive methods, requires 51.3% of their construction cost, and reduces estimation error by 16.4% on average and out-of-distribution estimation error by 21.8% on average. It was validated on over 70 text, multimodal and agent benchmarks, and its code and data are publicly available on GitHub. m:…2230

The fourth preprint looks at routing rather than scoring. Set-valued routing, which sends a request to a set of agents rather than picking a single best one, is superior to top-1 selection for compositional requests in multi-agent systems, the authors argue. On a 3,000-prompt benchmark based on WildChat, fine-tuned text encoders achieved 89.59 F1, outperforming GPT-4o at 41.51 F1. Weighted Agent Routing allows deterministic adjustment of the quality-cost operating point without retraining. The pattern is consistent with the rest of the week: the expensive general model is not always the best tool for the meta-task. m:…2820

What this means for builders

Four preprints in one week, all pointing at the measurement layer rather than the models, is a signal worth taking seriously. The field has spent two years arguing about which model tops which leaderboard, while the methods for building those leaderboards, judging quality, deciding restarts and routing work were quietly accumulating their own errors. These papers suggest the errors are large: a rubric that misranks quality, a monitor that looks accurate and loses points, a benchmark subset that misleads, a router that picks the wrong agent. m:…2832 m:…1316 m:…2230 m:…2820

None of this means evaluation is impossible. It means evaluation is a research problem in its own right, and teams that treat it as plumbing will get plumbing-grade answers. The practical takeaway is to test your evals the way you test your models: with held-out cohorts, with decision-level metrics rather than accuracy alone, and with a healthy suspicion of any single number that claims to summarise quality. m:…1316 m:…2832