Only 11% of developer turns ask for code. The benchmarks test 100%.

What developers actually do

AI21 Labs ran seven weeks of real developer traffic through an internal task classifier and the results should embarrass the benchmark industry. "In our traffic, only about 11% of real developer turns ask the agent to write code." The single most common message is a clarifying question. Confirmations, redirections, git commands and pasted context make up about 40% of real turns, "yet appear in no coding benchmark." Real messages average around 80 characters, arriving mid-session; over 60% of sessions contain more than one distinct task. m:…1076

The cost of the mismatch is concrete. AI21's intelligent gateway routes requests between models against a budget. Tested on a public benchmark, it "misjudged the traffic and ended up sending nearly 50% of the requests to the premium model, overstepping its target budget by 23 points." On a new 613-task benchmark built to mirror real traffic distribution, it landed 0.5 points from target. The benchmark you choose is a routing decision with a price tag. m:…1076

The texture of real usage is what benchmarks miss. AI21's messages average around 80 characters — quick, mid-session bursts, not the elaborate problem statements benchmarks are built from. To make its 613-task benchmark realistic, the company "mined our own internal repositories for tasks that had already been solved and merged." and set quotas from traffic distribution: "If 40% of our real traffic consisted of clarifying questions and codebase analysis, then 40% of our benchmark needed to test exactly that." m:…1076

Task flows, not issues

A preprint analyzing 4,782 coding-agent sessions from JetBrains IDEs makes the structural point: real usage is planning, refactoring, and review — flows that static, issue-derived benchmarks don't capture. Their SWE-TaskFlow method re-weights benchmarks to match a target interaction distribution. Along the way they measured something operators should know: sequential multi-step solving roughly doubles agent costs "without a stable increase in resolution rates." How you orchestrate the interaction matters more than which model you pick. m:…0083

The throughline: developers use agents as thought partners, in AI21's phrase, and benchmarks test them as code generators. Any team buying or routing models on SWE-bench-style numbers is optimizing for 11% of the job. m:…1076 m:…0083

SWE-TaskFlow's approach is calibration rather than replacement: transform existing benchmarks to match a target interaction distribution. The point isn't that issue-derived benchmarks are useless — it's that an uncalibrated benchmark is a random number generator with good marketing. m:…0083

The statistics are shaky too

Even where benchmarks do measure the right thing, the numbers may not mean what they claim. A study of 15,840 judged runs across 16 models found the median variance ceiling for repeated runs is just 1.34x — re-running buys you at most a 23% narrower confidence interval. Twelve of fifteen adjacent leaderboard pairs were statistically unresolved. Standard pooled-binomial intervals are, the authors write, about 2x too narrow for generalizing to new problems and 2x too wide for the fixed set. Leaderboard gaps that look decisive are often noise. m:…7136

And sometimes the labels are just wrong. A study of bug-inducing-commit detection found Claude Code beating ten SZZ baselines at F1 0.771 — then discovered the evaluation datasets contained incorrect oracle labels that deflated the reported metrics. The benchmarks aren't only misaligned with the work; in places they're mislabeled about the work they do test. m:…0869