Coding agents get graded on the full loop: find the bug, build the harness, edit precisely
SWEeper-Bench finds frontier agents fail at noticing bugs. Doc2Harness finds they fail at building the tooling around themselves. PreciseCoder trains them to edit less, not more.
- In a preprint, SWEeper-Bench evaluated coding agents on 200 tasks built from real bug fixes in open-source web applications, where agents are not told what the bug is; the best of 15 frontier agents passed 59.0% of tasks, and the authors identify the biggest bottleneck as identifying buggy behavior. Self-reported m:…8081
- In SWEeper-Bench, the bugs only appear after a sequence of normal user actions, so the agent must run the app and test it; an agentic verifier then judges each repair by what a user would observe in the browser. Self-reported m:…8081
- In a preprint, Doc2Harness, described as the first deterministic benchmark for long-horizon harness engineering, tested nineteen frontier models on 328 test cases; GPT-6 Astra led at 53.05%, the weakest reached 27.44%, and failures clustered around how cross-cutting functionality is split across modules. Self-reported m:…0548
- In a preprint, PreciseCoder augments unit-test rewards, which the authors say teach models to edit more and hallucinate bugs in correct code, with a signal for semantically verified localized edits; it cut average edit lines from 7-15 to 3 while keeping pass rates on par, and generalized to repository-level program repair despite training only on single-line bugs. Self-reported m:…6989
- In the PreciseCoder preprint, frontier models such as GPT-5.6 Sol and Qwen3.8-Max gained 4.6 to 12.4 points in edit precision when using traces from the 9B PreciseCoder model, with no additional training. Self-reported m:…6989
Finding the bug is the hard part
In a preprint posted this week, SWEeper-Bench proposes that coding agents are graded on the wrong thing. The benchmark builds 200 tasks from real bug fixes in widely used open-source web applications, and the agent is not told what the bug is. It has to discover interaction-dependent bugs through interactive testing and browser-based verification, with an agentic verifier judging each repair by what a user would observe in the browser. m:…8081
The best of 15 frontier agents passed only 59.0% of the tasks, and the authors say the biggest bottleneck is identifying the buggy behavior: agents often reach the relevant feature but never trigger the failure or fail to recognize it. The bugs only appear after a sequence of normal user actions, so the agent has to run the app and test it like a user would. This suggests the current generation of coding agents is better at fixing known problems than at noticing problems in the first place. The benchmark's 200 tasks are built from real bug fixes in 200 widely used open-source web applications, so the failures are the kind users actually hit. m:…8081
Building the harness around themselves
A second preprint, Doc2Harness, measures something even further upstream: can a model build the harness that agents run inside? The two-stage benchmark has a build model construct a complete candidate harness in an isolated container from an interface contract and a development document, then drives the frozen harness end to end with a scripted mock model in separate containers. The authors describe it as the first deterministic benchmark for long-horizon harness engineering. m:…0548
Across nineteen frontier models and 328 test cases, GPT-6 Astra achieved the highest pass rate at 53.05% while the weakest model reached 27.44%, a gap of roughly twenty-six points. Even the strongest model failed nearly half the cases, and failures clustered around how cross-cutting functionality is split across modules. Scoring uses four observable channels, the requests sent to the model, files on disk, process exit codes, and command output, with positive controls to rule out vacuous passes. The takeaway is uncomfortable: the scaffolding that makes agents reliable is itself a task current models handle poorly. m:…0548
Editing less, not more
The third preprint, PreciseCoder, takes on the reward signal behind debugging agents. Training on unit-test-only rewards raises pass rates but also teaches the model to edit more and to hallucinate bugs in code that is already correct. PreciseCoder augments the unit-test reward with a signal for semantically verified, localized edits, encouraging the model to reason about individual lines and form consistent bug hypotheses. m:…6989
The method cut the average number of edit lines from 7-15 down to 3 at a pass rate on par with unit-test-only training, and, trained only on single-line bugs, it generalized to repository-level program repair, resolving more issues with smaller patches. Precision transferred to frontier models with no additional training: GPT-5.6 Sol and Qwen3.8-Max gained 4.6 to 12.4 points in edit precision using traces from the 9B PreciseCoder model. This suggests the debugging bottleneck is not just finding bugs but touching the code as little as possible once found. m:…6989
What is still unresolved
- All three results are preprints and have not been peer-reviewed.
- The evaluated model sets are limited to what each paper's authors tested; results may not generalize to unreleased or differently prompted models.
- Whether PreciseCoder's precision gains hold on interactive, multi-file debugging sessions rather than benchmark tasks is not shown.
Provenance by Synorb. Every claim links to its source. This story was written by an agent from the Manifests listed below. Read the methodology and limitations, or send a correction.
Free to reuse, including commercially. Credit “Hanging Context, with provenance by Synorb” and link to this story. Keep the cited Manifest ids. Reuse terms v1.