The small model that can't do the job can still grade it

The fast brain gets hired

The most expensive part of running an agent is often not the agent. It is the judge that grades it. A study from Scale Labs this week argues that small decision models — System-1 models, built to answer rather than to reason at length — can replace generative LLM judges for many evaluation tasks. The researchers report System-1 judges are statistically indistinguishable from GPT-5.6 without reasoning on all five pairwise and best-of-4 benchmarks they tested, and that with one reference answer per prompt, eleven System-1 judges reach 90.3 to 93.4 percent on JudgeBench, level with frontier reasoning judges. Best-of-16 selection using System-1 judges costs about one-sixteenth as much as GPT-5.6 with reasoning. The judge, it turns out, does not need to be smart. It needs a reference answer. m:…3965

A second preprint explains why the small models work, and how to build them properly. NESTONE, a method for turning large language models into efficient System One decision models, starts from the finding that decision information in LLMs is concentrated in intermediate layers — the final six layers can be removed without performance loss. Independently trained smaller models largely mimic the full model rather than developing specialised decision capabilities, the authors report, so NESTONE jointly trains a full model with prefix sub-models, sharing final layers and decision heads, with a lightweight gate that reads the hidden state at each exit to decide whether more computation is needed. One model, several depths, chosen on the fly. The implication is that the decision layer of an agent system can be a small, fast model that knows when to stop thinking. m:…2592

The teacher who can't do

The same idea extends from judging to teaching. A third preprint shows that decision models that perform poorly as independent actors can still provide valuable feedback when judging completed trajectories — the JEV method. Judging whether each step helped is more effective for separating useful from useless actions than predicting final task success, and judging an action after the episode ends reveals consequences that acting must anticipate. The method improved a Qwen2.5-1.5B policy from 72.8 to 82.9 percent on ALFWorld and from 56.8 to 71.4 percent on WebShop. A model that cannot do the job can still grade the work and train the worker. That is a strange property, and a useful one: the teacher does not have to be better than the student at the task, only better at recognising a good step when it sees one. m:…2584

Cheap, accurate — and unreliable

The week's caution comes from SO-Arena, a benchmark for System One models across 11 domains and five capability families that measures decision reliability, not just accuracy: stability under controlled option-set changes, and recognition of missing answers. The findings are uncomfortable. Scaling and reasoning gains are inconsistent across domains, and strong overall performance does not imply reliable decision-making — the best-performing System One model in accuracy ranked seventh in reliability. A model can be right most of the time and still be the wrong judge, because the errors it makes are the ones that change the decision. m:…2581

The Scale Labs study adds a second warning: resistance to text that addresses the grader varies from 0 to 49 percent of correct verdicts reversed across different System-1 models. Some judges can be talked out of a correct verdict half the time. The authors' advice is practical rather than alarmed: select models based on public leaderboards, which track judging accuracy with a Pearson correlation of 0.95 to 0.96, and use confidence-based routing to escalate uncertain items to stronger reasoning models. The judging layer is becoming a product with its own failure modes, its own leaderboards, and its own routing rules — which is what happens when a capability gets cheap enough to use everywhere. m:…3965