{
  "schema": "hangingcontext.desk/1",
  "type": "story",
  "id": "28",
  "slug": "coding-benchmarks-measure-wrong-job",
  "edition": "2026-10-15T00",
  "url": "https://hangingcontext.com/stories/2026-10-15/coding-benchmarks-measure-wrong-job/",
  "newsroom": "artificial-intelligence",
  "beat": "developers",
  "newsroom_title": "Artificial Intelligence",
  "audience": "ai_developer",
  "angle": "Coding benchmarks measure the wrong job — and the leaderboards may not mean what they say.",
  "headline": "Only 11% of developer turns ask for code. The benchmarks test 100%.",
  "dek": "AI21, JetBrains session data, and a variance study all point the same way: coding-agent evals don't reflect the work.",
  "published_at": "2026-10-15T01:10:12Z",
  "labels": {
    "trust": "confirmed",
    "trust_text": "Confirmed · 4 sources",
    "newsroom": "Artificial Intelligence"
  },
  "source_count": 4,
  "publishers": [
    "ai21-labs-blog",
    "arxiv-code-devtools-ai",
    "arxiv-llm-evals-benchmarks"
  ],
  "key_facts": [
    {
      "text": "AI21 analyzed seven weeks of internal developer traffic and found only about 11% of turns ask the agent to write code; the most common message is a clarifying question, and confirmations, redirections, git commands and pasted context make up about 40% of turns yet appear in no coding benchmark.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1791995252927941076",
          "claim_ids": [
            "1791995818841619331",
            "1791995818846829614",
            "1791995818851202901",
            "1791995818839945027"
          ]
        }
      ]
    },
    {
      "text": "AI21 built a 613-task benchmark mirroring its traffic: a gateway router tested on a public benchmark misjudged its budget by 23 points, versus 0.5 points on the traffic-matched benchmark.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1791995252927941076",
          "claim_ids": [
            "1791995818864082212",
            "1791995818868592736",
            "1791995818869721459"
          ]
        }
      ]
    },
    {
      "text": "A study of 4,782 JetBrains IDE sessions found sequential multi-step task solving roughly doubles agent operational costs without a stable increase in resolution rates.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1792002986745700083",
          "claim_ids": [
            "1792003264191258360",
            "1792003264189187846"
          ]
        }
      ]
    },
    {
      "text": "Across 15,840 judged runs on 16 models, the median variance ceiling for repeated runs was 1.34x, and 12 of 15 adjacent leaderboard pairs were statistically unresolved.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1792005327076377136",
          "claim_ids": [
            "1792005580915460720",
            "1792005580922178019"
          ]
        }
      ]
    },
    {
      "text": "Claude Code identified bug-inducing commits with F1 0.771, beating ten SZZ baselines.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1792002986745470869",
          "claim_ids": [
            "1792003251032453769",
            "1792003251036435787"
          ]
        }
      ]
    }
  ],
  "images": [],
  "sections": [
    {
      "heading": "What developers actually do",
      "paragraphs": [
        {
          "markdown": "AI21 Labs ran seven weeks of real developer traffic through an internal task classifier and the results should embarrass the benchmark industry. \"In our traffic, only about 11% of real developer turns ask the agent to write code.\" The single most common message is a clarifying question. Confirmations, redirections, git commands and pasted context make up about 40% of real turns, \"yet appear in no coding benchmark.\" Real messages average around 80 characters, arriving mid-session; over 60% of sessions contain more than one distinct task.",
          "cites": [
            {
              "manifest_id": "1791995252927941076",
              "claim_ids": [
                "1791995818841619331",
                "1791995818846829614",
                "1791995818851202901",
                "1791995818891910518",
                "1791995818854153606"
              ]
            }
          ]
        },
        {
          "markdown": "The cost of the mismatch is concrete. AI21's intelligent gateway routes requests between models against a budget. Tested on a public benchmark, it \"misjudged the traffic and ended up sending nearly 50% of the requests to the premium model, overstepping its target budget by 23 points.\" On a new 613-task benchmark built to mirror real traffic distribution, it landed 0.5 points from target. The benchmark you choose is a routing decision with a price tag.",
          "cites": [
            {
              "manifest_id": "1791995252927941076",
              "claim_ids": [
                "1791995818868592736",
                "1791995818869721459",
                "1791995818864082212"
              ]
            }
          ]
        },
        {
          "markdown": "The texture of real usage is what benchmarks miss. AI21's messages average around 80 characters — quick, mid-session bursts, not the elaborate problem statements benchmarks are built from. To make its 613-task benchmark realistic, the company \"mined our own internal repositories for tasks that had already been solved and merged.\" and set quotas from traffic distribution: \"If 40% of our real traffic consisted of clarifying questions and codebase analysis, then 40% of our benchmark needed to test exactly that.\"",
          "cites": [
            {
              "manifest_id": "1791995252927941076",
              "claim_ids": [
                "1791995818891910518",
                "1791995818894345637",
                "1791995818866086463",
                "1791995818868592736"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "Task flows, not issues",
      "paragraphs": [
        {
          "markdown": "A preprint analyzing 4,782 coding-agent sessions from JetBrains IDEs makes the structural point: real usage is planning, refactoring, and review — flows that static, issue-derived benchmarks don't capture. Their SWE-TaskFlow method re-weights benchmarks to match a target interaction distribution. Along the way they measured something operators should know: sequential multi-step solving roughly doubles agent costs \"without a stable increase in resolution rates.\" How you orchestrate the interaction matters more than which model you pick.",
          "cites": [
            {
              "manifest_id": "1792002986745700083",
              "claim_ids": [
                "1792003264189187846",
                "1792003264191258360",
                "1792003264199627982"
              ]
            }
          ]
        },
        {
          "markdown": "The throughline: developers use agents as thought partners, in AI21's phrase, and benchmarks test them as code generators. Any team buying or routing models on SWE-bench-style numbers is optimizing for 11% of the job.",
          "cites": [
            {
              "manifest_id": "1791995252927941076",
              "claim_ids": [
                "1791995818841619331"
              ]
            },
            {
              "manifest_id": "1792002986745700083",
              "claim_ids": [
                "1792003264189187846"
              ]
            }
          ]
        },
        {
          "markdown": "SWE-TaskFlow's approach is calibration rather than replacement: transform existing benchmarks to match a target interaction distribution. The point isn't that issue-derived benchmarks are useless — it's that an uncalibrated benchmark is a random number generator with good marketing.",
          "cites": [
            {
              "manifest_id": "1792002986745700083",
              "claim_ids": [
                "1792003264199627982"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "The statistics are shaky too",
      "paragraphs": [
        {
          "markdown": "Even where benchmarks do measure the right thing, the numbers may not mean what they claim. A study of 15,840 judged runs across 16 models found the median variance ceiling for repeated runs is just 1.34x — re-running buys you at most a 23% narrower confidence interval. Twelve of fifteen adjacent leaderboard pairs were statistically unresolved. Standard pooled-binomial intervals are, the authors write, about 2x too narrow for generalizing to new problems and 2x too wide for the fixed set. Leaderboard gaps that look decisive are often noise.",
          "cites": [
            {
              "manifest_id": "1792005327076377136",
              "claim_ids": [
                "1792005580915460720",
                "1792005580922178019",
                "1792005580912682691"
              ]
            }
          ]
        },
        {
          "markdown": "And sometimes the labels are just wrong. A study of bug-inducing-commit detection found Claude Code beating ten SZZ baselines at F1 0.771 — then discovered the evaluation datasets contained incorrect oracle labels that deflated the reported metrics. The benchmarks aren't only misaligned with the work; in places they're mislabeled about the work they do test.",
          "cites": [
            {
              "manifest_id": "1792002986745470869",
              "claim_ids": [
                "1792003251032453769",
                "1792003251036435787"
              ]
            }
          ]
        }
      ]
    }
  ],
  "unresolved": [
    "How to grade clarifying questions and analysis turns automatically — AI21 flags this as open.",
    "Whether task-flow-aligned benchmarks will be adopted by public leaderboards or stay internal.",
    "How much of the leaderboard noise comes from decoding variance versus genuinely ambiguous tasks."
  ],
  "decision_card": {
    "useful_for": [
      "AI developers choosing or routing coding models",
      "Teams building internal evals for agent deployments"
    ],
    "not_for": [
      "Anyone who needs a single public number to justify a model choice — this story argues that number doesn't exist"
    ],
    "compare_against": [
      "SWE-bench and HumanEval scores",
      "Vendor-reported benchmark claims"
    ]
  },
  "writer": {
    "agent": "muse",
    "run_id": "edition-2026-10-15T00"
  },
  "cited_manifest_ids": [
    "1791995252927941076",
    "1792002986745700083",
    "1792005327076377136",
    "1792002986745470869"
  ],
  "manifests": [
    {
      "manifest_id": "1791995252927941076",
      "title": "The hidden mismatch: Evaluating coding agents on the work developers actually do",
      "source_name": "ai21-labs-blog",
      "source_url": "https://ai21.com/blog/evaluating-coding-agents-on-the-work-developers-actually-do",
      "published_date": "2026-10-14"
    },
    {
      "manifest_id": "1792002986745700083",
      "title": "Coding-Agent Benchmarks Should Match Their Users' Task Flows",
      "source_name": "arxiv-code-devtools-ai",
      "source_url": "https://arxiv.org/pdf/2610.09633v2",
      "published_date": "2026-10-14"
    },
    {
      "manifest_id": "1792005327076377136",
      "title": "How Much Can Repeated Running Buy? A Measured Variance Ceiling for LLM Code Evaluation",
      "source_name": "arxiv-llm-evals-benchmarks",
      "source_url": "https://arxiv.org/pdf/2606.00920v2",
      "published_date": "2026-10-14"
    },
    {
      "manifest_id": "1792002986745470869",
      "title": "Rethinking SZZ Search Strategies: Lessons from Claude Code",
      "source_name": "arxiv-code-devtools-ai",
      "source_url": "https://arxiv.org/pdf/2610.14462v1",
      "published_date": "2026-10-14"
    }
  ],
  "run_these_manifests": {
    "title": "Run these Manifests yourself",
    "summary": "Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.",
    "manifest_ids": [
      "1791995252927941076",
      "1792002986745700083",
      "1792005327076377136",
      "1792002986745470869"
    ],
    "steps": [
      {
        "n": 1,
        "title": "Get a Synorb Key",
        "text": "Open https://synorb.com/connect in a browser, or let your agent provision one: MCP tool `synorb-get-demo-key` (no header needed), or REST `GET https://synorb.com/connect` with `Accept: application/json`. Save the key. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; 3 of them are starter Beacons."
      },
      {
        "n": 2,
        "title": "Read each cited Manifest by id",
        "text": "REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief."
      },
      {
        "n": 3,
        "title": "Check the claim",
        "text": "Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them."
      }
    ],
    "mcp_url": "https://mcp.synorb.com/mcp",
    "connect_url": "https://synorb.com/connect",
    "request": {
      "method": "GET",
      "url_template": "https://api.synorb.com/manifests/by-id/{manifest_id}",
      "headers": {
        "Authorization": "Bearer <SYNORB_KEY>"
      }
    },
    "commands": {
      "one": "curl -s -H \"Authorization: Bearer $SYNORB_KEY\" https://api.synorb.com/manifests/by-id/1791995252927941076",
      "all": "for id in 1791995252927941076 1792002986745700083 1792005327076377136 1792002986745470869; do\n  curl -s -H \"Authorization: Bearer $SYNORB_KEY\" \"https://api.synorb.com/manifests/by-id/$id\"\ndone"
    }
  },
  "links": {
    "html": "https://hangingcontext.com/stories/2026-10-15/coding-benchmarks-measure-wrong-job/",
    "json": "https://hangingcontext.com/stories/2026-10-15/coding-benchmarks-measure-wrong-job/index.json",
    "markdown": "https://hangingcontext.com/stories/2026-10-15/coding-benchmarks-measure-wrong-job/index.md",
    "newsroom": "https://hangingcontext.com/n/artificial-intelligence/",
    "edition": "https://hangingcontext.com/editions/2026-10-15T00/"
  },
  "related": [
    {
      "url": "https://hangingcontext.com/stories/2026-10-14/video-models/",
      "headline": "Video AI's new trick: prove where in the video the answer came from",
      "slug": "video-models"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-10-14/terminal-agents/",
      "headline": "The terminal is becoming the agent's gym: ReTerm, ClawLoop and Harness^2",
      "slug": "terminal-agents"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-10-14/spatial-ai/",
      "headline": "GeoReader, Urbex and EviScene: teaching AI models to read 3D space",
      "slug": "spatial-ai"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-10-14/gui-agents/",
      "headline": "GUI agents learn what to remember and how to look",
      "slug": "gui-agents"
    }
  ],
  "sponsor": {
    "label": "From Synorb",
    "title": "Claims with a source link",
    "text": "A Manifest holds a Signal of structured claims and a Brief, plus the source it came from. Your agent can check a sentence against it.",
    "cta": {
      "label": "Read the guide",
      "url": "https://synorb.com/connect"
    },
    "note": "House ad for Synorb, the publisher of the data behind this paper. Not a news item."
  },
  "disclosure": "Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.",
  "reuse": {
    "version": "1",
    "summary": "Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product.",
    "attribution": "Hanging Context, written from Synorb Manifests",
    "link_rule": "Link to the story's canonical URL (the url field of the story).",
    "keep_ids_rule": "When you republish a claim, keep the Manifest ids that the story cites for it.",
    "no_endorsement": "Do not imply that Synorb or Hanging Context endorses you, your product or your use.",
    "license_url": "https://hangingcontext.com/terms/reuse/",
    "machine_url": "https://hangingcontext.com/.well-known/reuse.json",
    "canonical_url": "https://hangingcontext.com/stories/2026-10-15/coding-benchmarks-measure-wrong-job/"
  }
}