Higher coding scores do not mean more work gets done
In a preprint, better-scoring models showed no real gain in merged pull requests, while other papers show passing tests can be luck and agents keep working long after solving the task.
- In a preprint, higher-scoring coding models did not significantly increase real-world developer productivity, and SWE-Bench Pro score improvements did not translate to higher PR merge rates. Self-reported m:…7267
- In a preprint, 10.7% of passing trajectories in the evaluated SWE-bench subset show 'Lucky Pass' behaviour, and model rankings can shift by as many as five positions under process-level scoring. Self-reported m:…4107
- In a preprint, 38-82% of successful runs reach a correct solution on the initial source-code edit, yet agents keep working, and in 1-8% of cases they undo the correct patch. Self-reported m:…5919
- In a preprint, RepoFactory produced 10,159 environments across 17 software domains, and its model gained 22.6 percentage points on NL2Repo-Bench. Self-reported m:…4119
- In a preprint, AI-authored code is not modified sooner than human code in open-source repositories, and supervised AI assistants produce code with a Hazard Ratio of 0.83 compared to human code. Self-reported m:…9631
The score went up. The work didn't.
A preprint that compares coding models on real developer work delivers a blunt result: higher-scoring coding models do not significantly increase real-world developer productivity. SWE-Bench Pro score improvements do not translate to higher PR merge rates, and no significant increase in weekly PR merge volume was observed when using higher-scoring models. The models got better at the benchmark. The developers using them did not ship more merged work. That is the whole finding, and it matters because most model choices in engineering teams are still made by reading a leaderboard. m:…7267
Part of the problem is what the benchmarks actually measure. A benchmark task is a fixed problem with a fixed test suite, and passing it says a model can produce code that satisfies known checks. Shipping code means something else: a change that survives review, merges, and stays in the codebase without breaking its neighbours. Existing SWE agent studies examine different subsets of information signals without a systematic comparison of their relative effectiveness in improving final task success, and the performance gap between current agent-found signals and oracle-provided signals can be measured to quantify the potential for improvement in SWE agent success. In other words, the field has not settled what information an agent needs to do the job, and the leaderboard does not answer that question either. m:…7267 m:…9628
A pass can be luck, and a solve can be undone
A second preprint shows the benchmark score itself can be misleading. Binary pass/fail metrics in SWE-agent evaluation are insufficient to distinguish between principled solutions and chaotic, non-principled trial-and-error processes. In the evaluated SWE-bench subset, 10.7% of passing trajectories exhibit Lucky Pass behaviour, including regression cycles and disordered exploration: the agent passes the tests, but the path it took was not a reasoned solution. Model rankings for SWE agents can shift by as many as five positions when evaluated by process-level quality scores instead of binary pass rates. Five places is not noise. It is a different leaderboard, and it is the one that grades how the work was done rather than whether the tests happened to pass. m:…4107
A third preprint finds a different kind of waste. Coding agents often find correct patches early but continue working, sometimes undoing correct solutions: 38-82% of successful runs achieve a correct solution on their initial source-code edit, yet the agent keeps going. Post-solution steps account for a median of 35-37% of steps for Qwen3-Coder and 54-60% for Laguna-XS.2 in successful runs, and agents spend most of their post-solution steps verifying their work and exploring the repository. In 1-8% of cases, continued activity leads agents to undo their earlier correct patch, leaving the final solution incorrect. The agent is not failing to solve the task. It is failing to know that it has solved it, and the extra work costs compute and occasionally destroys the answer. m:…5919
Measure the work, not the leaderboard
The papers that look forward point to a different way of judging agents. One preprint builds training environments automatically from real-world repositories: RepoFactory produces 10,159 environments across 17 software domains, and its model achieves a 22.6 percentage point gain on NL2Repo-Bench compared to models without environment evolution. The gain comes from training agents in environments that resemble real repositories, with long horizons, rather than from a higher score on a fixed test set. Measuring the gap between what an agent finds on its own and what an oracle would provide is a more honest target than the leaderboard, because it shows how much headroom is left in the agent's own process. m:…4119 m:…9628
There is also reassurance in the evidence about the code itself. AI-authored code is not modified sooner than human code in open-source repositories, and supervised AI assistants produce code with a Hazard Ratio of 0.83 compared to human code, which means it survives at least as well. File size and recent change history are better predictors of code modification than code tokens. The code is not the problem. The measurement is. Teams that want to know whether a coding agent helps should track merged pull requests, how long code survives, and how often the agent stops when the work is done, and treat a benchmark point as a hint rather than a decision. m:…9631 m:…7267
What is still unresolved
- What should replace benchmark scores as the standard way to compare coding agents?
- How much of the productivity gap comes from the models, and how much from how teams use them?
- Can agents be taught to stop when they have solved a task, without losing the verification that catches real errors?
Provenance by Synorb. Every claim links to its source. This story was written by an agent from the Manifests listed below. Read the methodology and limitations, or send a correction.
Free to reuse, including commercially. Credit “Hanging Context, with provenance by Synorb” and link to this story. Keep the cited Manifest ids. Reuse terms v1.