Coding Agents Are Being Graded on the Wrong Tests, New Research Finds

Agents find different bugs than maintainers fix

In a preprint, researchers introduced PROBE, a benchmark of 2,457 bugs fixed over 18 months across 20 diverse GitHub repositories, designed to test proactive bug discovery by frontier coding agents. The results are sobering for anyone buying coding agents on headline scores: agents achieved recall below 21% and precision below 9% against the reference set of human-fixed bugs. Stronger models increased recall from 8% to 17% but did not improve precision, which suggests that scaling model capability helps agents find more of the right bugs without helping them avoid the wrong ones. The authors also report that 85% of agent-identified findings were judged valid by human evaluators even though precision against the reference set was low, and that bug distributions found by agents differ substantially from those fixed by human maintainers. m:…4214

The tension between those two findings is the point. Agents are finding real bugs, since most of their findings pass human judgment, but they are not finding the bugs maintainers prioritize. A team that adopts an agent expecting it to replicate maintainer triage will be disappointed, while a team that uses it as a second pair of eyes for a different class of issues may get value the headline precision figure hides. The preprint reframes the buying question from how good the agent is in general to which bug distribution it covers, and whether that distribution matches the team's backlog. m:…4214

Benchmark labels mislead

In a preprint, a second research team examined whether coding agent benchmarks actually resemble real software engineering work. The authors found that benchmark labels such as bug fixing and feature implementation are insufficient for determining whether evaluated work resembles a team's actual workload, and that every pair of benchmarks analyzed differs on at least two SNC axes, including benchmarks carrying the same task label. Agent behavior also varied: patch sizes and exploration patterns differed significantly between benchmarks even for identically labeled tasks, and resolved runs tended to involve less distributed changes and code with lower centrality values. m:…6974

For engineering leaders, the practical consequence is that a benchmark score is not portable across teams. Two teams buying on the same leaderboard number may be buying performance on very different kinds of work, and a vendor's best benchmark may be the one least like the buyer's codebase. The authors' remedy is granular: profile the work a team actually does, then choose benchmarks that mirror that profile, rather than trusting a label. m:…6974

Multi-stage work exposes the reliability gap

In a preprint, a third team built FeatureArc, a benchmark that requires coding agents to build features across multiple dependent stages, simulating real software evolution. Success rates fell from 62.7% at the first stage to 28.5% when the whole feature was tested on the final code. Inherited defects were present in 72.9% of failed later stages, and in stages following a failure, agents left the earlier error unnoticed about 95% of the time. The pattern is a failure to revisit and correct earlier work, which compounds across stages until the final feature test fails. m:…0491

FeatureArc's result connects directly to the other two preprints. If agents find a different distribution of bugs than maintainers, and if benchmark labels hide differences in task shape, then multi-stage reliability is where those gaps become expensive: an early defect that an agent never revisits becomes a late-stage failure. For founders building coding tools, the implication is to invest in verification and rework loops, not just generation quality, and for buyers, it is to test agents on multi-stage, evolving features from their own repositories rather than on single-shot tasks. All three preprints are early-stage research, and their numbers come from controlled benchmarks rather than production deployments, but together they give teams a sharper set of questions than any single leaderboard. m:…0491 m:…4214