Agents are now graded on running a store, shopping and driving robots

The test has moved to the job itself

Five preprints published on 11 and 12 November share one change of method: they grade agents on work that lasts, changes and pushes back, rather than on single answers. In a preprint, StoreBench puts an agent in charge of a simulated online apparel store for 30 to 45 simulated days, with 29 merchant tools to manage the store. In a preprint, CARTBench tests shopping advice on 169 timestamped cases across 17 retail domains, where requests are complex and ambiguous and come with timelines. m:…3438 m:…0363

The same shift runs through the other three preprints. In a preprint, GTAGym is an environment for evaluation and reinforcement learning on GTA V tasks, with 103 online tasks reconstructed across nine task families and 47 image-based shooting and sniping tasks. In a preprint, Embodied First Exam is a framework for coding agents that control robots, with 1,156 tasks, 35 suites and 47 robots. In a preprint, Interactive Visual Gym enables closed-loop evaluation of visual interaction models by jointly simulating task environments and human users, with 315 interaction scenarios across cooking, household manipulation and mobile-interface domains. m:…4270 m:…3102 m:…0362

The scores are low where the work is hardest

In the StoreBench preprint, Claude Fable 5.2 is reported as the leading model, with a mean composite score of 0.835 and an 89% pass rate. That result is the benchmark's own finding, not a product launch, and the preprint also reports a smart-triage baseline at a mean composite score of 0.764 with a 97% pass rate, so the leading model scores higher while passing less often. The preprint names underbuying as a recurring failure: it appears in 178 of 693 runs in the main suite, and incomplete refunds and unresolved returns can lead to escalating fines. m:…3438

The other preprints show the same pattern of partial success. In the CARTBench preprint, three of the ten models evaluated exceeded 90% on Core criteria, but none exceeded 45% on Excellence criteria, and the authors describe an understanding-to-execution gap between required behaviour and high-quality purchase advice. In the GTAGym preprint, the best configuration of frontier models reached 70.9% overall accuracy, while aerial navigation, evasion and interception remain difficult. In the Embodied First Exam preprint, GPT-6 Astra in Codex is the strongest of eight agents tested, with a 34% success rate on 456 core tasks in the Standard setting. In the Interactive Visual Gym preprint, Gemini 4 Argon completes only 53.4% of tasks within the time limit. m:…0363 m:…4270 m:…3102 m:…0362

How the tests try to stay honest

Each preprint describes controls meant to stop agents gaming the test. In the StoreBench preprint, agents manage a catalog of 3,448 product variants and source inventory from a market of 40 suppliers, and the benchmark evaluates agents with safeguards against reward hacking. StoreBench uses Medusa v2 as the underlying commerce platform for the simulated store. In the GTAGym preprint, the environment uses automatic engine-state outcome checks to verify task completion, rather than relying on a human to judge whether a game task was finished. m:…3438 m:…4270

The robot and visual tests add settings that change what the agent is allowed to know. In the Embodied First Exam preprint, the Standard setting provides only sensor observations and does not allow environment resets by default, while the Realtime setting keeps the environment running while the agent reasons, and the Privileged setting provides access to the full simulator state and unlimited resets. Coding agents in the framework can choose between direct control, code-to-control, or training a policy to control. In the Interactive Visual Gym preprint, the interaction-quality metric shows 78.5% agreement with human participant preferences, and closed-loop training using GRPO improves task success and interaction quality for the Qwen3.5-4B model in cooking environments. m:…3102 m:…0362

What a developer should take from this

Taken together, the five preprints argue for matching the test to the job rather than trusting a single leaderboard. A store test that runs for weeks exposes restocking and refund failures that a short test would miss, and a shopping test that separates Core from Excellence criteria shows that getting the basics right is not the same as giving expert advice. The GTAGym preprint also reports a concrete training result: LoRA reinforcement learning of Qwen3.8-27B on generated on-foot navigation tasks improved performance on native GTA V tasks from 15% to 23%. m:…3438 m:…0363 m:…4270

The limits are part of the finding. None of the five preprints claims agents have mastered these jobs: the best reported rates are 89% passing a store test, 70.9% on a game benchmark, 53.4% completing visual interaction tasks, and 34% on core robot tasks. For a team choosing an agent for real work, the useful question the preprints support is narrower and more practical: which failure modes does the benchmark expose, and do they match the failures that would cost money or time in the team's own setting? m:…3438 m:…4270 m:…0362 m:…3102