Coding agents are failing the test, not the task

The tests themselves are suspect

In a preprint, nearly 60% of unsolved SWE-bench Verified instances contain flawed tests. That single finding reframes a large part of the coding-agent leaderboard debate: when an agent fails an instance, the failure may belong to the test rather than to the model. Teams that rank models on these scores without asking about test quality are, on this evidence, ranking noise alongside capability. m:…6745

SWE-Bench ProMax, also in a preprint, is built to make that problem harder to hide. Its tasks average 11.5 modified files and 261.6 lines of code per instance, and the best frontier model achieves only a 41.2% resolve rate on it. The point of a larger, messier task is not cruelty for its own sake. A small, tidy task lets a model succeed by pattern matching; a task spread across many files forces it to hold a design together while it changes it. m:…6745

Give the agent a structure, not a wish

Skeleton coding, in a preprint, takes a different approach to the same problem: treat program structure as the specification. The paper reports that skeleton coding reduces mean test error by 48.4% relative to the best end-to-end behavioral specifications, and reduces implementation cost by 15 to 26 times. It also narrows the average gap between small and large models by about 52.4%. The implication is worth pausing on. If a better specification shrinks the gap between model sizes, then part of what large models were being paid for was compensating for vague instructions. m:…8241

This is a practical shift for anyone building coding tools. Instead of asking a model to infer an entire implementation from a behavioral description, hand it the skeleton: the shape of the program, with the pieces to fill in. The preprint's results suggest the agent then has less room to drift, and the reviewer has something concrete to check. Specification work, long treated as overhead, becomes the highest-leverage part of the pipeline. m:…8241

What developers actually do with generated code

A second preprint grounds the discussion in observed behavior rather than benchmark scores. The DECODE dataset contains 53.6K real-world in-IDE code edits sourced from more than 1,000 developers, and its headline finding is uncomfortable for tool vendors: developers frequently abandon AI-generated code due to high editing effort. Acceptance rate, the metric most demos optimize, misses the point if the accepted code is then rewritten or thrown away. m:…9808

The same preprint reports that frontier language models are prone to over-editing and reverting developer-written edits when attempting to infer intent, and that fine-tuning improves edit prediction similarity to developers but does not fully bridge the gap to frontier models. Put plainly, the model that writes code fluently is not automatically the model that edits it respectfully. An agent that overwrites a developer's deliberate choices has not helped, even if its replacement passes a test. m:…9808

Simpler methods, honestly measured

A systematic review and controlled re-evaluation, in a preprint, adds a skeptical note about the whole improvement literature. It finds that most code-improvement techniques do not provide gains over a budget-matched sampling baseline, and that best-of-n sampling with an analyzer-weighted ranker matches or beats complex techniques like planning, self-refinement, and agentic feedback. When the comparison controls for how many samples each method gets, much of the apparent progress evaporates. That is a warning about evaluation design, not just about any one technique. m:…9553

The review also reports that correctness losses on HumanEval+ are likely due to benchmark contamination rather than model degradation, which is the same lesson from a different angle: a benchmark that has leaked into training data stops measuring capability and starts measuring memorization. A separate preprint offers a constructive version of the fix, reporting that fine-tuning a coding model on data filtered by ExLabel results in a 66.2% resolution rate on SWE-bench Verified, a 14.5% improvement over the best baseline, and that existing SWE-bench-style datasets suffer from quality issues that limit the reliability of coding-agent evaluation. Filter the data, fix the test, specify the structure, and only then argue about the model. That ordering is the week's real consensus. m:…9553 m:…1475