The long-horizon report card: agents still fail exploration and expert-grade work
Two new benchmarks say the quiet part out loud: agents can't proactively explore changing environments, and none matches human experts on real paid operations work. A third paper proposes how to train past the failure.
- AdaExploreBench, published October 20, finds current LLM agents struggle to seek relevant evidence in evolving environments and detect stale knowledge; in experiments across eight leading LLMs, GPT-6 Sol achieved only 61.9% calibrated accuracy, and providing sufficient evidence improves average performance by close to 10%. (author-reported results) Self-reported m:…8104
- OR-License, published October 20, is a benchmark of 31 instances across 10 scenarios from paid operations-research engagements totaling USD 3.68 million; across 11 frontier LLMs, none matches expert-level feasibility and deployment-adjusted quality — the strongest Claude Code configuration reached 88.39% feasibility and a DAQ of 0.87 vs the expert reference of 1. (author-reported results) Self-reported m:…9415
- Interlock, published October 20, synthesizes long-horizon tasks as typed completion graphs to enforce cross-component constraints and filters trajectories with an agent-as-judge; fine-tuning Qwen3.6 backbones on accepted trajectories improved AutomationBench pass rates by 4.00–4.50 percentage points. (author-reported results) Self-reported m:…7792
Can't explore
The first benchmark, AdaExploreBench, published October 20, tests what happens when the environment changes without telling the agent. The authors instantiate environmental dynamics as hidden rules — including implicit rule drift, where the latent rule changes without announcement — and ask whether agents can actively discover those rules and adapt to unannounced changes. Experiments were conducted on eight leading LLMs. m:…8104
The verdict, in the authors' words: the benchmark reveals "that current models struggle to seek relevant evidence and detect stale knowledge." GPT-6 Sol achieved only 61.9% calibrated accuracy. The nuance matters more than the score: the authors find that agents "can often reason effectively from informative evidence once it is available" — the failure is in acquiring the evidence, not using it. Providing sufficient evidence improves average performance by close to 10%. Exploration, not reasoning, is the bottleneck. m:…8104
Can't match the expert
The second benchmark aims at expert-grade operations work rather than exploration. The authors present OR-License, an end-to-end benchmark of 31 instances across 10 industrial scenarios drawn from paid operations-research engagements totaling 3.68 million USD — real client work, not toy problems, with a controlled client simulator that discloses business facts without supplying OR expertise. The metric is deployment-adjusted quality, which averages expert-relative quality across instances and assigns zero to infeasible deliveries. m:…9415
The finding, per the authors: across 11 frontier LLMs tested with coding-agent harnesses, several beat the expert reference on conditional objective quality, but none matched its deployment-adjusted quality. The strongest primary Claude Code configuration achieved 88.39% strict feasibility and a DAQ of 0.87 against the expert reference of 1. Trajectory analysis identifies persistent formulation and implementation failures, with task-equalized research-chain completion rates ranging from 24% to 45% across models. The agents can optimize — they cannot yet practice the discipline like a professional. m:…9415
A fix with typed graphs
The third paper, Interlock, offers the constructive response. "Existing agents may satisfy individual requirements while failing to maintain these relationships, producing locally correct but incomplete outcomes." Interlock synthesizes long-horizon tasks as typed completion graphs to enforce cross-component constraints — grounding edges connect public evidence to decisions, coupling hyperedges impose joint constraints across components, and realization edges connect decisions to all required downstream effects. m:…7792
The graph compiles into a resettable workspace, a public request, and an evaluation rubric. Then, as the authors put it, "An agent-as-judge inspects execution artifacts and runs verification scripts, retaining trajectories judged to satisfy all mandatory requirements." Fine-tuning Qwen3.6 backbones on those accepted trajectories improved AutomationBench pass rates by 4.00–4.50 percentage points. It is a training-data strategy as much as an architecture: generate tasks with real constraints, keep only the completions that hold together. m:…7792
The executive takeaway
The pattern across the three papers is worth more than any single number. Both benchmarks locate the failure before the reasoning step — in exploration, evidence acquisition, and the formulation discipline that keeps a long task coherent. The fix follows the diagnosis: train on tasks that punish local correctness. For executives deploying agents in production-like settings, the operational read is to measure exploration and feasibility explicitly rather than trusting final-answer quality, because that is where the gap lives — and the gap is still wide. m:…8104 m:…9415 m:…7792
What is still unresolved
- Whether AdaExploreBench's hidden-rule environments predict agent behavior in real enterprise tool environments.
- Whether OR-License's 31 instances generalize to operations domains outside the 10 reported industrial scenarios.
- Whether Interlock-trained models retain gains on benchmarks beyond AutomationBench, JobBench, and Harvey LAB.
- Whether the reported ~10% evidence-provision lift holds when available evidence is adversarial or noisy.
Provenance by Synorb. Every claim links to its source. This story was written by an agent from the Manifests listed below. Read the methodology and limitations, or send a correction.
Free to reuse, including commercially. Credit “Hanging Context, with provenance by Synorb” and link to this story. Keep the cited Manifest ids. Reuse terms v1.