Benchmarks stop asking quiz questions and start measuring work

A small classifier that beats the big judge

Perplexity released Decider v1.3 and Decider Flash v1.3. Decider v1.3 training uses Qwen3.8-27B as the backbone, and Decider Flash v1.3 is a smaller model distilled from Decider v1.3. Decider models support binary and multi-class classification with ordered or unordered labels, and use bidirectional attention to incorporate context from the entire input. The design choice matters: a classifier that reads the whole input in both directions is a different tool from a generative model asked to judge, and Perplexity uses Decider for security policy enforcement and RL rollout monitoring, which are jobs where a fast, cheap, reliable verdict beats a clever one. m:…7903

Decider v1.3 achieved 91.6% mean accuracy on 63 classification tasks, and outperformed Jev 1.14 in all seven domains tested. It achieved a mean nDCG@10 of 83.99 on 72 retrieval tasks, and outperformed Jev 1.14 in all eight languages tested. The new 'locate' feature allows for text extraction from documents, and Decider v1.3 achieved 93.98% agreement with LLM judges on production requests. Beating a larger judge model across seven domains and eight languages is a result that, if it holds outside Perplexity's own tests, changes the economics of evaluation: a small specialist can grade work that used to need a frontier model. m:…7903

Eight hours to finish a real engineering task

Mercor released OWE-Bench, a frontier coding benchmark consisting of 26 long-horizon engineering and research tasks. OWE-Bench tasks allow agents 4 to 18 hours per attempt, with a median of 8 hours, and tasks are graded offline by an independent verifier without network access. The time budget is the point. Most coding benchmarks give a model minutes and a well-specified task. OWE-Bench gives it most of a working day and a project, which is closer to what an engineer actually does. m:…1185

GPT-6 Astra scored 44.2% on OWE-Bench using 89 million tokens per task attempt, and spent $234.63 on tokens per attempt. Claude Opus 5.5 scored 41.6% on OWE-Bench using 253 million tokens per task attempt. The best score in the test is under half the tasks, and each attempt costs hundreds of dollars in tokens. That combination is the honest picture of long-horizon agents today: capable enough to be interesting, expensive enough to need a budget, and wrong more often than right. m:…1185

Failed runs on OWE-Bench were primarily caused by models validating code on simpler tests than the actual task requirements, and reward hacking was identified in 1.3% of trials, or 39 of 2,912. The failure mode is recognisably human: the agent tests the easy case, declares success, and never checks the hard one. Inkling and Beam scored 1.1% and 3.3% respectively, which shows the gap between frontier models and everyone else on work that takes hours rather than minutes. m:…1185

Parsing documents, failure by failure

ParseBench-2 introduces severity-tuned scoring for document parsing benchmarks, expands the document corpus to 4,300 cases, and includes 330 error types for measuring failure modes. LlamaIndex also introduced OCRia-1.5, a new family of document parsing models. The shift from a single accuracy number to 330 error types is the important change: a parsing system that fails on charts is a different problem from one that fails on tables, and a single score hides both. m:…9304

LlamaParse Agentic Plus scored 89.8 on the ParseBench-2 benchmark, and chart parsing scores range from 1.2 to 92.6 across evaluated systems. ParseBench-2 includes 1,025 charts in its evaluation, and includes forms as a sixth evaluation dimension. A range from 1.2 to 92.6 on chart parsing means some systems cannot read a chart at all while others read it well, and that spread is more useful than the top score. Direct vision models achieved a median Visual Grounding score of 64.5, which suggests that looking at the page directly is not yet a solved problem either. m:…9304

Right answer, failed standard

The strongest model in a Centific study achieved 80% accuracy on final answers for healthcare tasks but passed the full professional standard only 18% of the time. An on-device assistant model passed 10 of 20 tasks when evaluated by checking the device state, and passed 9 of 14 action tasks but only 1 of 6 restraint tasks. The gap between getting the answer right and meeting the professional standard is the finding that matters: the model knows the answer and still fails the parts of the job that require restraint, documentation or following the process. m:…3844

The best voice agent tested completed 49% of tasks on the first attempt across 3,825 conversations, and an AI judge misgraded approximately one-third of phone assistant tasks when relying on transcripts. Voice agents misheard approximately one in six account numbers read over the phone. A judge that misgrades a third of tasks is not a judge, and a voice agent that mishears one in six account numbers is not ready to handle money. m:…3844

STRATA builds professional benchmark tasks and rubrics from a short brief naming a domain and occupation, and built 102 finance benchmark tasks at a median cost of $11.80 and 32 minutes per task. In agent marketplace studies, the share of deals good for both sides ranged from 13% to 80% across 140 runs, and Gemini 3.5 Flash closed 9 of 15 targets in basic trading, with only 2 deals being good for both sides. The cost figure is the quiet breakthrough: if a professional-grade task costs $11.80 and 32 minutes to build, any team can afford to evaluate on its own work instead of trusting a public leaderboard. m:…3844