Coding agents stumble when the workspace shifts and the client stays vague

Static benchmarks flatter real tools

Most coding benchmarks hold the world still while the agent works. Real repositories do not. Dependencies change, requirements move, and sessions restart. A preprint posted October 26 builds a benchmark around that reality, testing coding agents against mid-execution workspace changes such as shifting dependencies or requirements. The result is a sharp downgrade: agents show lower success rates in shifted workspaces than in static ones, and the gap is large rather than marginal. m:…6192

The headline numbers make the point concretely. One frontier model's success rate falls from 67.35% to 24.79% when the workspace shifts. And when the shift involves tool or session changes combined with stale evidence, the benchmark reports zero paired success across all 17 evaluated systems. That last figure is the one to sit with: under that combination, no system solved both the original and the shifted version of the task. The failure is not about one model's weakness. It is about how agents handle evidence that has gone out of date. m:…6192

Why does a shifting workspace hurt so much? Because an agent's plan rests on observations that are no longer true: a dependency version it already read, a file layout it already mapped, a requirement it already confirmed. When the environment changes without notice, the agent keeps reasoning from a snapshot of a world that no longer exists. The benchmark's contribution is to make that failure measurable instead of anecdotal, by shifting the workspace mid-execution rather than between runs. m:…6192

The client problem

A second preprint tests a different real-world gap: the brief itself. Its benchmark uses 100 tasks drawn from real freelance software jobs, each worth at least $4,000, and grades agents both on clarifying incomplete requirements through multi-round interaction and on delivering working software. The clarification side is measured against 512 questions paired with reference answers from real client-freelancer exchanges, so the benchmark knows what a good question looks like. The best reported result, a 67% task pass rate for one frontier model in independent delivery, shows that even the strongest current systems fail about a third of realistic freelance jobs. m:…7071

Read together, the two benchmarks describe the actual job. An agent must pin down what the client wants, then keep working while the ground moves. Current evaluations that test either skill in isolation will overstate readiness for work that demands both. m:…6192 m:…7071

Planning beats wandering

The third preprint offers a counterweight. Instead of letting a model navigate a repository iteratively, burning tokens on each step, its method plans first: select candidate files, form hypotheses, and only then retrieve source. That planning-based approach to repository-level fault localization cuts token usage by 90.3% compared with iterative methods, and reaches 59.3% top-function accuracy on a verified coding benchmark, ahead of two named alternatives at 53.8% and 56.2%. The authors' claim is that planning before retrieval beats model-controlled wandering, on both accuracy and cost. m:…3692

The lesson across all three papers is about where agent effort goes. Systems that re-plan when evidence changes, ask the clarifying question early, and decide before they retrieve will waste less and survive longer. Systems that assume the workspace and the brief will stay still are being graded on a world that does not exist. m:…6192 m:…3692