Give an AI agent 24 hours and the leaderboard changes
Five new benchmarks measure long-horizon work, memory governance, visual evidence and exploration. The best models lead by wide margins — and the tests say short evaluations mislead.
- In Infrathon, 90% of agent runs (135 of 150) achieved further improvements during the second half of their 24-hour budget, in a preprint. Self-reported m:…0713
- On GameReplica, Claude Opus 4.8 scored 71.6% overall, while other models scored between 4.0% and 43.0%, in a preprint. Self-reported m:…2473
- VISTAQA reports a substantial gap between current models' answer accuracy and their visual evidence alignment, in a preprint. Self-reported m:…2471
- HEAMS reports that recall and retention governance policies improve agent memory resistance to poisoning, but recall policies often decrease helpfulness, in a preprint. Self-reported m:…0707
What changed
Five benchmarks share one complaint: the way agents are usually tested does not match the way they are used. The tests that matter now run for hours, ask the agent to remember things safely, demand evidence for what it claims to see, and check whether it can explore something unfamiliar before it starts writing code. The papers behind them report wide gaps between the best model and the rest, and they warn that short evaluations rank models differently than long ones do. All five are preprints, so none of the results has been peer reviewed. m:…0713 m:…0707 m:…2471 m:…2473 m:…2458
Time changes the ranking
Infrathon is the longest test in the batch. It benchmarks language agents on real-world, long-horizon infrastructure optimization, and its headline finding is that model rankings vary significantly depending on the evaluation horizon. In the paper's runs, 90% of agent runs, 135 of 150, achieved further improvements during the second half of their 24-hour budget. The implication is that a one-hour test and a one-day test are testing different things, and a ranking from the short test may not survive the long one. On one of the benchmark's tasks, Claude Opus 5.5 achieved a 19.86x KV-cache compression ratio, against a 6.82x human-designed baseline. m:…0713
Memory, evidence and exploration
GameReplica tests vision-language agents by asking them to replicate black-box systems. Claude Opus 4.8 scored 71.6% overall, while other models scored between 4.0% and 43.0%. The gap is not just size. Vision-language agents consistently score higher on visual fidelity than on rule consistency, the paper reports, which means the replicas look right before they behave right. Raising the difficulty from level 1 to level 5 caused a 33% to 63% performance decline in lower-scoring models, compared with less than 5% for the strongest model, and most failures of weaker models happened during the exploration phase, before any replica code was written. m:…2473
Two more benchmarks attack the same weakness from the evidence side. VISTAQA requires models to provide precise segmentation masks supporting their answers, and its GROVE metric combines textual accuracy and grounding quality via a geometric mean to enforce joint correctness. The paper reports that current models exhibit a substantial gap between answer accuracy and visual evidence alignment: they get the answer right more often than they can show their work. SciWorldBench applies the same logic to video generation, using paired scientific experiments with single-condition variations to expose superficial visual priors in video models. Current TI2V models frequently fail to distinguish between scientifically distinct outcomes in paired experiments, the paper reports, and it proposes independent vision-language-model-based evaluation without access to condition labels as a robust way to assess scientific accuracy in video generation. m:…2471 m:…2458
HEAMS turns to the part of agent work that benchmarks usually ignore: memory. Memory systems evaluated only on performance and efficiency fail to account for security, privacy, and user sovereignty risks, the paper argues. Governance policies, specifically recall and retention policies, improve agent memory system resistance to poisoning, it reports, but the same recall policies can reduce inappropriate information disclosure at the cost of helpfulness, often resulting in a decrease in agent helpfulness. The trade is the finding: a memory that never leaks is also a memory that helps less, and the benchmark's point is that both sides should be measured together. m:…0707
Why it matters
Taken together, the five benchmarks describe a testing regime catching up with how agents are actually deployed: over long sessions, with memory that persists, on tasks where looking right is not the same as being right. The practical consequence is that a single leaderboard number is becoming less useful, because the ranking depends on how long the test runs and what it measures besides accuracy. None of the five papers is peer reviewed, and all five numbers come from their authors' own runs, so the gaps they report are claims to check rather than settled facts. But the direction is consistent: the tests are getting longer, harder and more suspicious of a good-looking answer, and the models that lead do so by a lot. m:…0713 m:…0707 m:…2473 m:…2471 m:…2458
What is still unresolved
- All five benchmarks are preprints; their rankings and gaps are the authors' own results until independent groups reproduce them.
- Whether the long-horizon improvements Infrathon reports generalize beyond infrastructure optimization tasks is not shown.
- HEAMS leaves open how to set recall and retention policies that keep both safety and helpfulness, rather than trading one for the other.
Provenance by Synorb. Every claim links to its source. This story was written by an agent from the Manifests listed below. Read the methodology and limitations, or send a correction.
Free to reuse, including commercially. Credit “Hanging Context, with provenance by Synorb” and link to this story. Keep the cited Manifest ids. Reuse terms v1.