---
title: Give an AI agent 24 hours and the leaderboard changes
description: Five new benchmarks measure long-horizon work, memory governance, visual evidence and exploration. The best models lead by wide margins — and the tests say short evaluations mislead.
canonical: https://hangingcontext.com/stories/2026-11-18/agents-get-longer-harder-tests/
last-updated: 2026-11-18
url: https://hangingcontext.com/stories/2026-11-18/agents-get-longer-harder-tests/
published: 2026-11-18T00:49:50Z
edition: 2026-11-18T00
newsroom: research-science
beat: None
trust: reported
source_count: 5
cited_manifest_ids: ['1794916694104960713', '1794913302881942473', '1794913302881942471', '1794916694104960707', '1794913302879942458']
---

# Give an AI agent 24 hours and the leaderboard changes

*Five new benchmarks measure long-horizon work, memory governance, visual evidence and exploration. The best models lead by wide margins — and the tests say short evaluations mislead.*

Reported · 5 Manifests · Research & Science · Published 2026-11-18 00:49 UTC

## Key facts

- In Infrathon, 90% of agent runs (135 of 150) achieved further improvements during the second half of their 24-hour budget, in a preprint. (self-reported) [cites: 1794916694104960713]
- On GameReplica, Claude Opus 4.8 scored 71.6% overall, while other models scored between 4.0% and 43.0%, in a preprint. (self-reported) [cites: 1794913302881942473]
- VISTAQA reports a substantial gap between current models' answer accuracy and their visual evidence alignment, in a preprint. (self-reported) [cites: 1794913302881942471]
- HEAMS reports that recall and retention governance policies improve agent memory resistance to poisoning, but recall policies often decrease helpfulness, in a preprint. (self-reported) [cites: 1794916694104960707]

## What changed

Five benchmarks share one complaint: the way agents are usually tested does not match the way they are used. The tests that matter now run for hours, ask the agent to remember things safely, demand evidence for what it claims to see, and check whether it can explore something unfamiliar before it starts writing code. The papers behind them report wide gaps between the best model and the rest, and they warn that short evaluations rank models differently than long ones do. All five are preprints, so none of the results has been peer reviewed. [cites: 1794916694104960713, 1794916694104960707, 1794913302881942471, 1794913302881942473, 1794913302879942458]

## Time changes the ranking

Infrathon is the longest test in the batch. It benchmarks language agents on real-world, long-horizon infrastructure optimization, and its headline finding is that model rankings vary significantly depending on the evaluation horizon. In the paper's runs, 90% of agent runs, 135 of 150, achieved further improvements during the second half of their 24-hour budget. The implication is that a one-hour test and a one-day test are testing different things, and a ranking from the short test may not survive the long one. On one of the benchmark's tasks, Claude Opus 5.5 achieved a 19.86x KV-cache compression ratio, against a 6.82x human-designed baseline. [cites: 1794916694104960713]

## Memory, evidence and exploration

GameReplica tests vision-language agents by asking them to replicate black-box systems. Claude Opus 4.8 scored 71.6% overall, while other models scored between 4.0% and 43.0%. The gap is not just size. Vision-language agents consistently score higher on visual fidelity than on rule consistency, the paper reports, which means the replicas look right before they behave right. Raising the difficulty from level 1 to level 5 caused a 33% to 63% performance decline in lower-scoring models, compared with less than 5% for the strongest model, and most failures of weaker models happened during the exploration phase, before any replica code was written. [cites: 1794913302881942473]

> **From Synorb** — Connect your agent over MCP. Add https://mcp.synorb.com/mcp to your agent, then call the tool synorb-get-demo-key if it has no key yet. It needs no email and no form. Keys are free for 72 hours, and new keys are limited per network. [How to connect](https://synorb.com/connect/start)


Two more benchmarks attack the same weakness from the evidence side. VISTAQA requires models to provide precise segmentation masks supporting their answers, and its GROVE metric combines textual accuracy and grounding quality via a geometric mean to enforce joint correctness. The paper reports that current models exhibit a substantial gap between answer accuracy and visual evidence alignment: they get the answer right more often than they can show their work. SciWorldBench applies the same logic to video generation, using paired scientific experiments with single-condition variations to expose superficial visual priors in video models. Current TI2V models frequently fail to distinguish between scientifically distinct outcomes in paired experiments, the paper reports, and it proposes independent vision-language-model-based evaluation without access to condition labels as a robust way to assess scientific accuracy in video generation. [cites: 1794913302881942471, 1794913302879942458]

HEAMS turns to the part of agent work that benchmarks usually ignore: memory. Memory systems evaluated only on performance and efficiency fail to account for security, privacy, and user sovereignty risks, the paper argues. Governance policies, specifically recall and retention policies, improve agent memory system resistance to poisoning, it reports, but the same recall policies can reduce inappropriate information disclosure at the cost of helpfulness, often resulting in a decrease in agent helpfulness. The trade is the finding: a memory that never leaks is also a memory that helps less, and the benchmark's point is that both sides should be measured together. [cites: 1794916694104960707]

## Why it matters

Taken together, the five benchmarks describe a testing regime catching up with how agents are actually deployed: over long sessions, with memory that persists, on tasks where looking right is not the same as being right. The practical consequence is that a single leaderboard number is becoming less useful, because the ranking depends on how long the test runs and what it measures besides accuracy. None of the five papers is peer reviewed, and all five numbers come from their authors' own runs, so the gaps they report are claims to check rather than settled facts. But the direction is consistent: the tests are getting longer, harder and more suspicious of a good-looking answer, and the models that lead do so by a lot. [cites: 1794916694104960713, 1794916694104960707, 1794913302881942473, 1794913302881942471, 1794913302879942458]

## What is still unresolved

- All five benchmarks are preprints; their rankings and gaps are the authors' own results until independent groups reproduce them.
- Whether the long-horizon improvements Infrathon reports generalize beyond infrastructure optimization tasks is not shown.
- HEAMS leaves open how to set recall and retention policies that keep both safety and helpfulness, rather than trading one for the other.

## Decision card

- **Useful for:** Teams choosing models for long-running agent work, where short benchmarks may mislead; Builders of agent memory systems weighing safety against helpfulness
- **Not for:** Anyone looking for peer-reviewed rankings or a single leaderboard number to decide on
- **Compare against:** Short-horizon, accuracy-only benchmarks; Memory systems evaluated on performance and efficiency alone

## Provenance ledger

| Manifest id | Title | Source | Claims used | Published |
|---|---|---|---|---|
| `1794916694104960713` | Infrathon: Benchmarking Language Agents for Real-World, Long-Horizon AI Infra Optimization | [arxiv-model-efficiency-engineering](https://arxiv.org/pdf/2611.11995v2) | 1794917053476962202, 1794917053462962200, 1794917053478962204 | 11-17 00:00 |
| `1794913302881942473` | GameReplica: A Benchmark for Black-Box Visual Game Replication by Vision-Language Agents | [arxiv-multimodal-document-ai](https://arxiv.org/pdf/2609.22308v2) | 1794913667405945335, 1794913667406945336, 1794913667407945337, 1794913667408945338 | 11-17 00:00 |
| `1794913302881942471` | VISTAQA: Benchmarking Joint Visual Question Answering and Pixel-Level Evidence | [arxiv-multimodal-document-ai](https://arxiv.org/pdf/2605.20676v2) | 1794913665554944929, 1794913665537944927, 1794913665548944928 | 11-17 00:00 |
| `1794916694104960707` | HEAMS: Holistic Evaluation of Agent Memory Systems | [arxiv-model-efficiency-engineering](https://arxiv.org/pdf/2611.20144v1) | 1794917054167962364, 1794917054181962365, 1794917054146962363 | 11-17 00:00 |
| `1794913302879942458` | SciWorldBench: Do Video World Models Really Understand Scientific Experiments? | [arxiv-multimodal-document-ai](https://arxiv.org/pdf/2611.20899v1) | 1794913665380944822, 1794913665382944826, 1794913665384944830 | 11-17 00:00 |

## Run these Manifests yourself

Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.

1. **Get a Synorb Key.** Click Generate Free Agent Key at https://synorb.com/connect/start, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{"client":"your-agent-name"}`. The key is the `api_key` field of the answer. Save the key. Existing 72 Hour Agent Demo keys keep their issued limits. New keys require email OTP verification. An existing demo key lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons.
2. **Read each cited Manifest by id.** REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief.
3. **Check the claim.** Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them.

Cited Manifest ids:

- `1794916694104960713`
- `1794913302881942473`
- `1794913302881942471`
- `1794916694104960707`
- `1794913302879942458`

```sh
for id in 1794916694104960713 1794913302881942473 1794913302881942471 1794916694104960707 1794913302879942458; do
  curl -s -H "Authorization: Bearer $SYNORB_KEY" "https://api.synorb.com/manifests/by-id/$id"
done
```

> **From Synorb** — Save what you follow as a Beacon. A Beacon is a saved query that your agent pulls with an explicit lookback window. Every new account starts with 5 starter Beacons: Business & Technology Podcasts, AI Companies, Venture Capital, Private Companies and Public Company Filings & Executive Transitions. [See Beacons](https://synorb.com/connect/start)

## Related stories

- [Agents remember too much, too expensively. Four papers cut the memory bill.](https://hangingcontext.com/stories/2026-11-21/agent-memory-is-the-bottleneck/)
- [The chemistry lab is closing the loop — design, synthesis and measurement in one pass](https://hangingcontext.com/stories/2026-11-20/chemistry-labs-close-the-loop/)
- [Five preprints, one bottleneck: the race to shrink the memory bill for long-context models](https://hangingcontext.com/stories/2026-11-18/long-context-memory-squeeze/)
- [The score is not the skill: AI benchmarks keep flattering the models](https://hangingcontext.com/stories/2026-11-16/benchmarks-flatter-the-models/)

## Links

- HTML: https://hangingcontext.com/stories/2026-11-18/agents-get-longer-harder-tests/
- JSON: https://hangingcontext.com/stories/2026-11-18/agents-get-longer-harder-tests/index.json
- Markdown: https://hangingcontext.com/stories/2026-11-18/agents-get-longer-harder-tests/index.md

Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.

---
Reuse (terms v1): Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product. Credit: "Hanging Context, with provenance by Synorb", with a link to https://hangingcontext.com/stories/2026-11-18/agents-get-longer-harder-tests/. Terms: https://hangingcontext.com/terms/reuse/
