Coding benchmarks are done with single PRs: now it's multi-turn users, PR DAGs, pre-edit steering

The user, simulated

In a preprint this week, UserSWE argues that single-turn benchmarks for coding agents produce false positives and false negatives — they don't represent what happens when a real developer goes back and forth with an agent over several turns. Its fix: evaluate coding agents in multi-turn conversations with dynamic users. m:…6761

The headline result is that the leaderboard is budget-dependent: coding agent rankings change significantly based on the interaction budget allowed during task resolution. Claude Fable 5.1 leads in single-turn resolution, GPT-6-Astra leads at 8 turns, and Claude Opus 5 leads at intermediate budgets. Pick your agent by how many turns your workflow actually gives it, not by a single-turn score. m:…6761

PR history as a DAG

WEAVE takes a different cut at realism: real code evolution. The preprint's method weaves verified PR histories into dependency-structured tasks, organizing related PRs into bounded directed acyclic graphs to create unified executable tasks. Instead of scoring one PR in isolation, the benchmark models the way work actually accumulates — changes that depend on earlier changes. m:…6086

The measurement innovation is Graph Credit, a deterministic, test-grounded metric for dependency-consistent partial progress. The authors argue it gives more faithful progress measurement than flat scoring for long-horizon code agents — partial credit that respects the dependency graph, which also makes it usable for reinforcement learning on long-horizon tasks. m:…6086

Steer before the first edit

PAUSE is the most immediately usable of the four. Instead of searching or replanning mid-task, it injects a single repair plan before the coding agent's first source-code edit — search-free steering. The reported gains: resolution up 16.8% on SWE-bench Verified in OpenHands and 14.7% on Multilingual in SWE-agent, at 11 to 15% of the planner cost of the SICA baseline. m:…9251

It also works as a recovery tool: applied offline, PAUSE recovers 35.4% of eligible failed Verified tasks, and resuming from the recorded prefix uses 33 to 41% fewer executor tokens than restarting from scratch. For developers, that's a cheap intervention with a measurable payoff — plan once, before touching code. m:…9251

The long-horizon testbed

Agent4Sci widens the lens to science itself: a long-horizon testbed measuring how far today's agents can improve the published best models on 20 scientific-ML benchmarks across 11 domains. The sobering finding is that frontier agents improve results mainly through engineering tricks rather than domain insight — they climb the hill, but they don't understand the terrain. m:…6466

Trajectory analysis identified three recurring limitations: novelty miscalibration, resource under-utilization, and rapid collapse into local engineering sweeps. But there's a constructive result too: lightweight plug-in skills for research ideation, experiment management and scheduled checklists raised Claude Opus from 10.4% to 20.1% mean relative improvement, and Codex from 4.4% to 8.1%. Scaffolding, again, beats raw scale. m:…6466

Together, the four papers point one way: the field is done pretending a single PR diff is the unit of software engineering. The new unit is the multi-turn session, the dependency graph, the planned intervention — and the agents that thrive will be the ones evaluated the way they're actually used. For teams choosing a benchmark, the practical move is to match it to the deployment: multi-turn sims for assistant workflows, DAG-structured tasks for long-horizon autonomy, and pre-edit steering where token budgets are tight. m:…6761 m:…6086 m:…9251 m:…6466