Benchmarks say AI can't replace programmers yet -- but the power users are shipping anyway

The benchmarks say: not yet

Three papers landed this week with the same awkward message for the code-agent hype cycle. REPOCOD, a new benchmark built from 980 real-world tasks across 11 popular repositories, finds that none of the ten LLMs tested achieves more than 30% pass@1 -- and argues that existing benchmarks like CoderEval, DevEval, and RepoEval overestimate real-world coding ability through short completions, synthetic examples, or limited scale. One practical finding: retrieval-augmented generation beats handing the model the target function's dependencies as context, a detail that matters for anyone building repo-level agents. m:…2915

CETBench attacks from a different angle: it tests whether LLMs can tell if two programs are functionally equivalent after simple code transformations. The authors report significant performance drops on transformed pairs, evidence that current models lack deep semantic understanding of code. And a third preprint looks at code repair, finding frontier models routinely over-edit: high functional correctness masks unnecessarily large changes that raise cognitive complexity and maintenance burden. The fix is cheap and immediate -- preservation instructions in the prompt cut average excess Levenshtein distance from 0.195 to 0.131 and added cognitive complexity by 26.6% -- and the authors report reinforcement learning beats supervised fine-tuning for out-of-domain edit fidelity. All three are preprints, so treat the numbers as reported rather than settled. m:…6914 m:…3564

The practitioners say: ship anyway

Meanwhile, GitHub's Java Champions are not waiting for the benchmarks to catch up. Vlad Mihalcea built custom Copilot skills from 65 of his published articles so generated docs and code match his established principles and voice, and he uses the tool to design and implement features across his Hypersistence libraries -- cutting one new web application build from weeks to two days. Patrick Baumgartner has moved almost entirely to agentic workflows, migrated 900 slides from Reveal.js to Marp in eight to ten hours with visual and algorithmic verification, and explicitly asks the agent to interview him first to surface hidden assumptions. Frank Delporte assigned a real Pi4J bug -- virtual thread pinning in native calls -- to the Copilot coding agent with two file paths as a starting point. m:…4789

The shared discipline is the point. Delporte says all AI-generated pull requests in Pi4J were reviewed and approved by maintainers before merging, and he made 'add a unit test to validate the change' a standing instruction rule. Baumgartner runs a 'maintainer-audit' skill across his repositories to enforce project standards automatically. The model, in their framing, is a sparring partner: fast at execution, accountable to no one, and only as good as the human who verifies. That is a very different relationship than replacement -- and it survives every one of this week's benchmark results. m:…4789

The economics keep moving

The third force in the story is price. MiniMax's M3.1 Flash -- a multimodal mixture-of-experts model with 428B total parameters and 23B active, previously tested under the codename 'Space Bunny' -- is now free on the Kilo coding platform for a limited time. The company says agentic tasks cost 70% less overall than with its predecessor, MiniMax Sparse Attention cuts per-token compute at 1M context to roughly 1/20th of the previous generation, and list pricing is $0.15 per million input tokens and $0.60 per million output tokens -- about a fifth of Gemini 3.8 Flash's input price. m:…3184

Independent testing tempers the vendor story: Artificial Analysis gives M3.1 Flash an Intelligence Index of 42.7 and notes discrepancies with vendor-reported benchmarks. But the direction of travel is the developers' takeaway: agentic-capable models are getting cheaper faster than benchmarks are getting easier. If the models cannot replace programmers yet, the gap is being attacked from both sides -- and the practitioners winning today are the ones who built the verification habits, not the ones waiting for a benchmark to green-light them. m:…3184 m:…2915 m:…4789