The new agent benchmarks don't test chat. Frontier models are failing them

The eval frontier moved

Three preprints posted this week agree on one thing: the interesting benchmarks no longer ask whether a model can chat. They ask whether it can build something that works. CineRecon hands agents reference videos and asks for editable 3D scenes in Blender — 50 videos, 360 evaluation clips, 12 frontier models tested. ExeWorldBench goes further and asks for code-based 3D worlds that actually function: mechanisms that move, multi-stage systems with causal dependencies. And a psychometric audit asks whether LLMs can stand in for human survey respondents without embarrassing themselves. m:…5155 m:…9191

The results are uniformly deflating, and that is the news. In a preprint, the CineRecon authors report that models fail at action timing, contact relations and coordinated subject motion even under the least demanding representation setting. In a preprint, the ExeWorldBench authors report substantial limitations in executable world modeling across their three tasks. The message for anyone buying agent systems: the demo that wowed you was probably a chat-shaped task, and the benchmarks that matter now are shaped like physics. m:…5155 m:…9191

In a preprint, the CineRecon authors grade difficulty deliberately across three representation settings — Proxy, Simplified, Detailed. In a preprint, the ExeWorldBench authors do the same with three tasks: Mechanism Executability, which reconstructs individual mechanisms from visual observations; Compositional Executability, covering multi-stage systems with spatial, mechanical and causal dependencies; and Intent-World Synthesis, which builds executable worlds from high-level goals and physical requirements. The authors' thesis is that coding agents are evolving from reproducing what a world looks like toward constructing executable worlds that capture how a world works. m:…5155 m:…9191

Even the humans are hard to fake

The psychometric audit is the most uncomfortable of the three because it targets a use case people already pay for: synthetic survey respondents. In a preprint, the researchers tested 14 decision models and 47 generative LLMs against real human survey data — 263 respondent profiles, 68 items — and found that the best generative model, MiniMax M3, reached a Psychometric Similarity Score of 0.813 while a humble Gaussian copula fit in-sample scored 0.927. A classical statistics baseline beat every LLM tested. m:…0822

Worse, newer did not mean better: in 5 of 10 product lines, the newest release scored below its predecessor. The authors' conclusion is that generating believable survey responses is not the same as generating psychometrically valid ones — current synthetic respondents are fine for sample-level fidelity but not for predicting individual humans. If your team uses LLM panels for market research, this preprint says to check the validity math before the invoice. m:…0822

There is a nuance worth keeping: the paper does not say LLMs are useless as synthetic respondents. In a preprint, the authors report that 33 of 47 generative models reproduce item-correlation patterns better than any decision model — so for studying how answers hang together, they clear the bar. The failure is narrower and more important: predicting what any individual human would say, where the authors report every decision model doing worse than simply copying other respondents' answer frequencies. Use synthetic panels for distributions, never for persons. m:…0822

In a preprint, the audit authors add two details. First, format matters: probability vectors land closer to human item marginals than rounded answers, and profile regression beats copying other respondents' answer frequencies by only 1.0%. Second, 13 of the 14 purpose-built decision models scored below a generative reference, gpt-5.4-mini. If you pay a premium for a specialized survey model, this preprint suggests a generic LLM — or a copula — may do the job cheaper. m:…0822

What changes for buyers

The practical takeaway is about procurement, not research taste. Chat benchmarks are saturated enough that vendors can shop for flattering numbers; the new benchmarks are adversarial by construction — they test the exact things (timing, contact, causality, individual-level validity) that demos gloss over. An executive evaluating an agent vendor should ask which of these harder benchmarks the vendor has run, and treat 'we score well on chat evals' as a non-answer. m:…5155 m:…9191

All three are preprints, so treat the findings as directional rather than settled — but the direction is consistent across independent groups, which is more than you usually get in a single week. The eval conversation is moving from 'can it talk' to 'can it build, and can it be trusted to stand in for a person.' Budget your skepticism accordingly. m:…0822

The line worth remembering comes from the CineRecon authors: subject motions and interactions remain difficult to reproduce even under the Proxy setting. In a preprint, they frame the benchmark around reconstructing dynamic videos as editable Blender scenes, and argue the missing pieces — action timing, contact relations — are exactly what controllable video generation needs. For a buyer, that reframes the demo: the question is not whether it looks right, but whether you can change it — which suggests no current model can do it yet. m:…5155