Coding agents finally get graded on the hard parts

The signal matters more than the model

ORACLE-SWE asks an unfashionable question: what actually determines whether a software-engineering agent succeeds? The preprint's answer is intermediate information signals — reproduction tests, regression tests, edit location, execution context, API usage. These, the authors argue, are the primary determinants of success on repository-level tasks. And there is a measurable gap between the signals agents extract on their own and oracle-provided signals. The bottleneck is not reasoning; it is evidence gathering. A coding agent that cannot find the right reproduction test is a brilliant developer locked out of the repo. m:…3404

The same agents are also, it turns out, paranoid. ParanoiaEval benchmarks unnecessary defensive work in agentic coding and finds it in 11.2 to 58.7 percent of runs — despite explicit evidence that the caution was unneeded. Stronger task capability does not ensure more appropriate risk treatment; risk judgment is, the authors argue, an independent capability dimension. Your best coder and your best judge of when to be careful are two different skills, and scaling one does not scale the other. m:…7159

Memory without the crutch

Long-horizon coding is where agents go to forget. OPRD — on-policy reminder distillation — trains agents with reminders and then takes the crutch away: the student model operates without reminders at inference time. The preprint reports an 11.3-point improvement in benchmark-averaged task resolution over the open-source Qwen3.5-4B base model, and a cut in aggregate error events from 446 to 222, a 50.2 percent reduction. Train with training wheels, ship without them. The inference budget notices the difference. m:…1231

Solo beats the committee

RefactorPlatform, an open-source harness for repository-scale refactoring agents, delivers two results that cut against the multi-agent hype. First, AST-aware chunking beats naive token-window chunking by 25 to 30 percentage points on refactoring tasks — how you cut the code up matters more than how smart the model is. Structure beats tokens. m:…5544

Second, lean RAG-based single agents beat multi-agent delegation configurations 86 percent to 66 percent on matched refactoring tasks — and retrieval-augmented refactoring agents hit cost-neutrality per successful task despite the extra token overhead. The committee is more expensive, slower, and worse. For repo-scale refactoring, at least, the evidence says: one good agent with good retrieval beats a team of them. m:…5544

The builder's checklist

If you maintain a code RAG pipeline, the RefactorPlatform numbers are a direct instruction: parse the code, do not just window it. AST-aware chunking's 25-to-30-point advantage over naive token windows is the largest single delta in this week's set, and it comes from respecting the structure the programmer wrote. Retrieval quality is a parsing problem before it is a ranking problem. m:…5544

Two more items for the checklist. First, grade risk judgment as its own dimension: ParanoiaEval's finding that it is independent of task capability means your best-performing agent may also be your most pointlessly cautious one, burning budget on defensive work the evidence did not call for. Second, steal OPRD's trick — train with reminders, ship without them. An 11.3-point gain and halved error events for zero inference-time cost is the rare free lunch; the catch is only that you have to build the reminder curriculum first. m:…7159 m:…1231