---
title: The long-horizon report card: agents still fail exploration and expert-grade work
description: Two new benchmarks say the quiet part out loud: agents can't proactively explore changing environments, and none matches human experts on real paid operations work. A third paper proposes how to train past the failure.
canonical: https://hangingcontext.com/stories/2026-10-20/the-long-horizon-report-card-agents-still-fail-exploration-and-expert/
last-updated: 2026-10-20
url: https://hangingcontext.com/stories/2026-10-20/the-long-horizon-report-card-agents-still-fail-exploration-and-expert/
published: 2026-10-20T13:03:07Z
edition: 2026-10-20T12
newsroom: artificial-intelligence
beat: developers
trust: reported
source_count: 3
cited_manifest_ids: ['1792486813260688104', '1792486813260029415', '1792486813260917792']
---

# The long-horizon report card: agents still fail exploration and expert-grade work

*Two new benchmarks say the quiet part out loud: agents can't proactively explore changing environments, and none matches human experts on real paid operations work. A third paper proposes how to train past the failure.*

Reported · 3 sources · Artificial Intelligence · Published 2026-10-20 13:03 UTC

## Key facts

- AdaExploreBench, published October 20, finds current LLM agents struggle to seek relevant evidence in evolving environments and detect stale knowledge; in experiments across eight leading LLMs, GPT-6 Sol achieved only 61.9% calibrated accuracy, and providing sufficient evidence improves average performance by close to 10%. (author-reported results) (self-reported) [cites: 1792486813260688104]
- OR-License, published October 20, is a benchmark of 31 instances across 10 scenarios from paid operations-research engagements totaling USD 3.68 million; across 11 frontier LLMs, none matches expert-level feasibility and deployment-adjusted quality — the strongest Claude Code configuration reached 88.39% feasibility and a DAQ of 0.87 vs the expert reference of 1. (author-reported results) (self-reported) [cites: 1792486813260029415]
- Interlock, published October 20, synthesizes long-horizon tasks as typed completion graphs to enforce cross-component constraints and filters trajectories with an agent-as-judge; fine-tuning Qwen3.6 backbones on accepted trajectories improved AutomationBench pass rates by 4.00–4.50 percentage points. (author-reported results) (self-reported) [cites: 1792486813260917792]

## Can't explore

The first benchmark, AdaExploreBench, published October 20, tests what happens when the environment changes without telling the agent. The authors instantiate environmental dynamics as hidden rules — including implicit rule drift, where the latent rule changes without announcement — and ask whether agents can actively discover those rules and adapt to unannounced changes. Experiments were conducted on eight leading LLMs. [cites: 1792486813260688104]

The verdict, in the authors' words: the benchmark reveals "that current models struggle to seek relevant evidence and detect stale knowledge." GPT-6 Sol achieved only 61.9% calibrated accuracy. The nuance matters more than the score: the authors find that agents "can often reason effectively from informative evidence once it is available" — the failure is in acquiring the evidence, not using it. Providing sufficient evidence improves average performance by close to 10%. Exploration, not reasoning, is the bottleneck. [cites: 1792486813260688104]

## Can't match the expert

The second benchmark aims at expert-grade operations work rather than exploration. The authors present OR-License, an end-to-end benchmark of 31 instances across 10 industrial scenarios drawn from paid operations-research engagements totaling 3.68 million USD — real client work, not toy problems, with a controlled client simulator that discloses business facts without supplying OR expertise. The metric is deployment-adjusted quality, which averages expert-relative quality across instances and assigns zero to infeasible deliveries. [cites: 1792486813260029415]

> **From Synorb** — Ask what Synorb covers. The MCP tool synorb-stream-search answers questions like "do you have this publisher?" without using any Manifest allowance. [Connect and ask](https://synorb.com/connect)


The finding, per the authors: across 11 frontier LLMs tested with coding-agent harnesses, several beat the expert reference on conditional objective quality, but none matched its deployment-adjusted quality. The strongest primary Claude Code configuration achieved 88.39% strict feasibility and a DAQ of 0.87 against the expert reference of 1. Trajectory analysis identifies persistent formulation and implementation failures, with task-equalized research-chain completion rates ranging from 24% to 45% across models. The agents can optimize — they cannot yet practice the discipline like a professional. [cites: 1792486813260029415]

## A fix with typed graphs

The third paper, Interlock, offers the constructive response. "Existing agents may satisfy individual requirements while failing to maintain these relationships, producing locally correct but incomplete outcomes." Interlock synthesizes long-horizon tasks as typed completion graphs to enforce cross-component constraints — grounding edges connect public evidence to decisions, coupling hyperedges impose joint constraints across components, and realization edges connect decisions to all required downstream effects. [cites: 1792486813260917792]

The graph compiles into a resettable workspace, a public request, and an evaluation rubric. Then, as the authors put it, "An agent-as-judge inspects execution artifacts and runs verification scripts, retaining trajectories judged to satisfy all mandatory requirements." Fine-tuning Qwen3.6 backbones on those accepted trajectories improved AutomationBench pass rates by 4.00–4.50 percentage points. It is a training-data strategy as much as an architecture: generate tasks with real constraints, keep only the completions that hold together. [cites: 1792486813260917792]

## The executive takeaway

The pattern across the three papers is worth more than any single number. Both benchmarks locate the failure before the reasoning step — in exploration, evidence acquisition, and the formulation discipline that keeps a long task coherent. The fix follows the diagnosis: train on tasks that punish local correctness. For executives deploying agents in production-like settings, the operational read is to measure exploration and feasibility explicitly rather than trusting final-answer quality, because that is where the gap lives — and the gap is still wide. [cites: 1792486813260688104, 1792486813260029415, 1792486813260917792]

## What is still unresolved

- Whether AdaExploreBench's hidden-rule environments predict agent behavior in real enterprise tool environments.
- Whether OR-License's 31 instances generalize to operations domains outside the 10 reported industrial scenarios.
- Whether Interlock-trained models retain gains on benchmarks beyond AutomationBench, JobBench, and Harvey LAB.
- Whether the reported ~10% evidence-provision lift holds when available evidence is adversarial or noisy.

## Decision card

- **Useful for:** Technology executives scoping agent deployments in production-like settings.; Teams designing agent evals that measure exploration and feasibility, not just final answers.; Researchers building long-horizon task generation pipelines.
- **Not for:** Model selection on coding or chat benchmarks — these benchmarks target exploration and long-horizon completion.; Anyone expecting a vendor comparison; all three are research papers, not product evaluations.
- **Compare against:** Each benchmark's own baselines: 8 leading LLMs on AdaExploreBench, 11 frontier LLMs on OR-License, Qwen3.6 backbones on Interlock.; Deployment-adjusted quality (expert reference = 1) rather than raw objective quality.

## Provenance ledger

| Manifest id | Title | Source | Claims used | Published |
|---|---|---|---|---|
| `1792486813260688104` | AdaExploreBench: Benchmarking LLM Agents for Proactive Exploration and Test-time Adaptation in Evolving Environments | [huggingface-daily-papers](https://huggingface.co/papers/2610.25950) | 1792487131530016558, 1792487131559085272, 1792487131563813524, 1792487131524097175, 1792487131528211567, 1792487131543894726, 1792487131551603147, 1792487131535990906, 1792487131566791620 | 10-20 00:00 |
| `1792486813260029415` | OR-License: Evaluating LLM Agents as Real-World Operations Research Practitioners | [huggingface-daily-papers](https://huggingface.co/papers/2610.24087) | 1792487131985520384, 1792487131993960557, 1792487131995270297, 1792487131988661907, 1792487131991543276, 1792487131997535789, 1792487132003190040 | 10-20 00:00 |
| `1792486813260917792` | Interlock: From Local Correctness to Task Completion in Long-Horizon Agents | [huggingface-daily-papers](https://huggingface.co/papers/2610.19686) | 1792487135319669826, 1792487135322876167, 1792487135342766722, 1792487135341283209, 1792487135327715436, 1792487135333576303 | 10-20 00:00 |

## Run these Manifests yourself

Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.

1. **Get a Synorb Key.** Click Generate Free Agent Key at https://synorb.com/connect, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{"client":"your-agent-name"}`. The key is the `api_key` field of the answer. Save the key. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; 3 of them are starter Beacons.
2. **Read each cited Manifest by id.** REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief.
3. **Check the claim.** Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them.

Cited Manifest ids:

- `1792486813260688104`
- `1792486813260029415`
- `1792486813260917792`

```sh
for id in 1792486813260688104 1792486813260029415 1792486813260917792; do
  curl -s -H "Authorization: Bearer $SYNORB_KEY" "https://api.synorb.com/manifests/by-id/$id"
done
```

> **From Synorb** — Run the cited Manifests yourself. Every claim in this paper points to a Manifest id. Get a Synorb Key and read the Manifests behind it. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; 3 of them are starter Beacons. [Get a key](https://synorb.com/connect)

## Related stories

- [Stop comparing GPU hourly rates: the new math of buying AI infrastructure](https://hangingcontext.com/stories/2026-10-20/stop-comparing-gpu-hourly-rates-the-new-math-of-buying-ai-infrastructu/)
- [Deepfake detection is winning the demo and losing the deployment](https://hangingcontext.com/stories/2026-10-20/deepfake-detection-race/)
- [Voice AI ships for production into a quality gap and a deepfake wave](https://hangingcontext.com/stories/2026-10-20/voice-ai-production/)
- [Trillion-vector search, SSD-speed index updates, and gated answers: the retrieval rebuild](https://hangingcontext.com/stories/2026-10-20/retrieval-rebuild/)

## Links

- HTML: https://hangingcontext.com/stories/2026-10-20/the-long-horizon-report-card-agents-still-fail-exploration-and-expert/
- JSON: https://hangingcontext.com/stories/2026-10-20/the-long-horizon-report-card-agents-still-fail-exploration-and-expert/index.json
- Markdown: https://hangingcontext.com/stories/2026-10-20/the-long-horizon-report-card-agents-still-fail-exploration-and-expert/index.md

Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.

---
Reuse (terms v1): Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product. Credit: "Hanging Context, with provenance by Synorb", with a link to https://hangingcontext.com/stories/2026-10-20/the-long-horizon-report-card-agents-still-fail-exploration-and-expert/. Terms: https://hangingcontext.com/terms/reuse/