Coding agents stumble on the unglamorous parts of real repo work

Work that spans repositories

In a preprint, researchers built CrossSWE to benchmark cross-repository joint development for coding agents. It comprises 171 real-world instances spanning 165 repositories, 20 technical domains and 13 programming languages. The headline finding is about consistency: cross-repository inconsistency is the most prevalent failure mode for coding agents, accounting for 44.2% of failures. The strongest model evaluated, GPT-6.1 Sol, achieves only 58.48% Pass@1 on the benchmark. This suggests the hard part of real development is not writing code in one place, but keeping changes consistent across places that depend on each other. m:…8380

Bugs the agent itself introduces

In a preprint on foresight, researchers ask whether coding agents catch their own bugs, and their answer is that they largely do not. Current coding agents exhibit low foresight resolve rates when tasked with avoiding future defects introduced by their own code changes, the preprint says. Foresight-SWE provides a standardized benchmark for measuring whether coding agents can anticipate and address unstated problems during software engineering tasks. The distinction matters: passing a stated test is not the same as noticing that a change will break something nobody asked about yet. This suggests review effort should focus on second-order effects of agent-written code, not just on whether the requested feature works. m:…1757

Read together, the cross-repo and foresight findings point to the same weakness: agents handle the task as written, not the system around it. Cross-repo inconsistency dominates failures, and foresight resolve rates are low, which suggests the missing skill is not coding speed but awareness of consequences elsewhere in the codebase. For a team, that reframes what to check: the review question is less whether the change does what was asked, and more what else it touches that nobody mentioned. m:…8380 m:…1757

Learning new work, forgetting old work

In a preprint on continual evolution, researchers ask whether coding agents can keep learning without forgetting. Evolution on a subsequent dataset can substantially undermine performance on previously seen datasets, the preprint says. The example is stark: DeepSeek-V4-Flash-0731 performance on SWE-Bench drops from 94.0% to 32.0% after evolution on FeatureBench. This suggests teams that keep fine-tuning an agent on new work should re-test old capabilities after each round, rather than assuming new skills add to old ones. A coding agent that improves on the latest dataset while losing two-thirds of its earlier benchmark performance is not clearly improving at all. m:…2107

Review and performance, the two weak spots

In a preprint on code review, CR-Bench-v2 evaluates the real-world utility of AI code review agents. Current AI code review agents detect only about 33% of known defects, the preprint says, and Gemini Code Assist detects 8.1% of known defects in the CR-Bench framework. Existing code review benchmarks rely on noisy text-similarity metrics that fail to measure precision, the preprint argues, which means the field may have been grading review quality with the wrong ruler. If detection sits near a third, human review is not an optional extra on agent-written code; it is the main safety net. m:…7392

In a preprint on performance work, SWE-Opt evaluates language models on performance-oriented, repository-level optimization. Current models struggle significantly with performance optimization, the preprint says, with runtime gains sparse and inconsistent and memory optimizations nearly non-existent. By contrast, expert-written optimizations achieve an aggregate speedup of 15.5x and peak memory reduction of 171.3x over the benchmark tasks. An agentic setting with iterative repository exploration improves patch correctness substantially but does not translate into performance gains. This suggests exploration helps agents write code that works, but not code that runs faster or leaner, and that performance engineering remains human work for now. m:…7550

The overall pattern across these five preprints is consistent rather than scattered: agents lose consistency across repositories, miss defects they introduce, forget earlier capabilities when they evolve, catch only a minority of known defects in review, and fail to turn correct patches into faster code. That pattern suggests the sensible division of labour for now is to let agents draft and explore, while humans own consistency, review and performance. The benchmarks do not say agents are useless; they say the unglamorous maintenance work is where human judgement still carries the weight. m:…8380 m:…1757 m:…2107 m:…7392 m:…7550