{
  "schema": "hangingcontext.desk/1",
  "type": "story",
  "id": "705",
  "slug": "coding-agents-measured-not-shipped",
  "edition": "2026-10-28T00",
  "url": "https://hangingcontext.com/stories/2026-10-28/coding-agents-measured-not-shipped/",
  "newsroom": "artificial-intelligence",
  "beat": "developers",
  "newsroom_title": "Artificial Intelligence",
  "angle": "Coding agents are entering their measurement phase: harder benchmarks, evaluated mods, and inspectable output",
  "headline": "Coding agents get measured: FrontierCode 2.0 tops out at 45 percent while teams learn to evaluate their own mods",
  "dek": "Cognition says Claude Opus 5.5 leads FrontierCode 2.0 at 45 percent, and Arize shows why agent mods need their own evaluation.",
  "published_at": "2026-10-28T00:51:21Z",
  "labels": {
    "trust": "confirmed",
    "trust_text": "Confirmed · 4 sources",
    "newsroom": "Artificial Intelligence"
  },
  "source_count": 4,
  "publishers": [
    "arize-ai-blog",
    "cline-blog",
    "dataiku-blog",
    "windsurf-blog"
  ],
  "key_facts": [
    {
      "text": "Cognition says Claude Opus 5.5 is the top-performing model on FrontierCode 2.0 with a score of 45 percent.",
      "cites": [
        {
          "claim_ids": [
            "1793134194425423450"
          ],
          "manifest_id": "1793133898238389637"
        }
      ],
      "self_reported": true
    },
    {
      "text": "Cognition says FrontierCode 2.0 tasks are drawn from 55 percent more repositories than FrontierCode 1.0, with thoroughness criteria growing 3x compared with FrontierCode 1.1.",
      "cites": [
        {
          "claim_ids": [
            "1793134194422000920",
            "1793134194427623500"
          ],
          "manifest_id": "1793133898238389637"
        }
      ],
      "self_reported": true
    },
    {
      "text": "Arize reports that Claude Code mods are small TypeScript functions that rewrite prompts, add UI or replace built-in features, and are not sandboxed, creating privacy or security risks.",
      "cites": [
        {
          "claim_ids": [
            "1793124875161784874",
            "1793124875164453799"
          ],
          "manifest_id": "1793124531092636500"
        }
      ],
      "self_reported": false
    },
    {
      "text": "TypeSafe AI says Jev is approximately 194 times faster and 445 times cheaper than traditional LLMs on its workflow evals.",
      "cites": [
        {
          "claim_ids": [
            "1793128757077683101",
            "1793128757079388748"
          ],
          "manifest_id": "1793128461355145256"
        }
      ],
      "self_reported": true
    }
  ],
  "images": [],
  "sections": [
    {
      "heading": "The benchmark gets harder",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1793134194425423450",
                "1793134194418776538"
              ],
              "manifest_id": "1793133898238389637"
            }
          ],
          "markdown": "The Cognition Team released FrontierCode 2.0 as an updated coding benchmark with stricter mergeability standards and broader task coverage, and the headline result is humbling: Claude Opus 5.5 is the top-performing model with a score of 45 percent. Agentic software engineering remains far from solved, the team says, and the number gives teams a realistic ceiling for what current models do on hard, real-world-style tasks."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1793134194422000920",
                "1793134194427623500",
                "1793134194424824346",
                "1793134194431100392"
              ],
              "manifest_id": "1793133898238389637"
            }
          ],
          "markdown": "The benchmark is harder in measurable ways. FrontierCode 2.0 tasks are drawn from 55 percent more repositories than FrontierCode 1.0, thoroughness criteria grew by 3x compared with FrontierCode 1.1, and tasks include 43 percent more blocking criteria than previous versions. In 80 percent of tasks, the reference solution differs from the upstream pull request, up from 60 percent in FrontierCode 1.0, which means models cannot coast by reproducing a known patch."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1793134194421695322",
                "1793134194428675035"
              ],
              "manifest_id": "1793133898238389637"
            }
          ],
          "markdown": "Models struggle to infer user intent, respect unstated constraints and anticipate long-term consequences of their changes, according to the benchmark authors, and models can score lower at higher reasoning effort levels on FrontierCode 2.0. That second finding is worth pausing on: more thinking does not automatically mean better code, and teams that pay for higher effort should check whether the extra tokens buy accuracy on their own repositories."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1793134194429765816"
              ],
              "manifest_id": "1793133898238389637"
            }
          ],
          "markdown": "One measure did improve: the failure rate for scope criteria in FrontierCode 2.0 is 3.8 percent, compared with 9.0 percent in FrontierCode 1.1. Scope discipline is getting better even as overall tasks get harder, which suggests models are learning to stay inside the task, even when they cannot yet finish it well."
        }
      ]
    },
    {
      "heading": "Your own mods need measuring too",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1793124875161784874",
                "1793124875166697463",
                "1793124875170237755"
              ],
              "manifest_id": "1793124531092636500"
            }
          ],
          "markdown": "Benchmarks for frontier models are only half the story, because teams now customize their coding agents. Arize describes Claude Code mods as small TypeScript functions that allow users to rewrite prompts, add UI or replace built-in features, and notes that Claude Code currently ships six built-in mods, five of which are enabled by default. One of them, the agents-md mod, was developed in response to user requests for Claude Code to read AGENTS.md files."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1793124875164453799",
                "1793124875172041169",
                "1793124875170237755",
                "1793124875169131282"
              ],
              "manifest_id": "1793124531092636500"
            }
          ],
          "markdown": "The caution in the Arize post is direct: Claude Code mods are not sandboxed, creating potential privacy or security risks, and a regex-based guard in a mod does not function as a secure sandbox. A ghost mod built to evaluate coding turns adds a median of 1.7 seconds to each turn, a small cost, but tracing agent sessions with Arize AX provides visibility into mod costs and execution without determining whether mod judgments are accurate."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1793124875173846746",
                "1793124875174642258",
                "1793124875176368663"
              ],
              "manifest_id": "1793124531092636500"
            }
          ],
          "markdown": "Evaluating agent mods requires a reference judge and a versioned dataset to distinguish between mod bugs and policy disagreements, Arize argues, and LLM judges must be provided with the same context as the agent being evaluated to avoid false negatives. A larger model does not necessarily perform better as a judge if it lacks the same input context as the agent. The lesson generalizes: a team that ships a custom agent modification without an evaluation harness is flying blind, however good the base model is."
        }
      ]
    },
    {
      "heading": "Faster decisions, inspectable output",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1793128757074173602",
                "1793128757080214245",
                "1793128757085372820",
                "1793128757077683101",
                "1793128757079388748"
              ],
              "manifest_id": "1793128461355145256"
            }
          ],
          "markdown": "TypeSafe AI introduced Jev, a decision model designed to address latency and cost challenges in traditional LLMs, and Cline has integrated it through plugins for browser navigation, prompt testing and tool-call guardrails. Jev classifies input states into choices, scores or probabilities instead of emitting tokens, and Jev-guard allows instant guardrails to judge tool calls before LLM execution. TypeSafe says Jev is approximately 194 times faster and 445 times cheaper than traditional LLMs on its workflow evals, figures that are company-reported and should be tested on a team's own decision tasks."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1793128757087112256",
                "1793128757082209513",
                "1793128757083487693"
              ],
              "manifest_id": "1793128461355145256"
            }
          ],
          "markdown": "Cline also offers ClinePass, a $9.99 per month subscription plan providing access to open-weight models like DeepSeek, Kimi, GLM and Qwen, alongside plugins Jev-browser, which provides a real browser inside Cline for automated navigation, and Jev-playground, which translates vague prompts into Jev requests via a web interface. The combination points to a division of labor: small decision models for structured choices, large models for language, and a subscription tier for open-weight experimentation."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1793112082015530105",
                "1793112082019724515",
                "1793112082025826631",
                "1793112082029321878",
                "1793112082018125369",
                "1793112082022545281",
                "1793112082020866844",
                "1793112082030535392"
              ],
              "manifest_id": "1793111719500630125"
            }
          ],
          "markdown": "Dataiku Headless closes the loop on the governance side, allowing developers to use coding agents to build and modify Dataiku projects, with Cobuild generating data pipelines, models and dashboards. It is available as a plugin for Claude Code, Codex, Cursor, Snowflake CoCo, AWS Kiro and OpenCode, can be connected using an MCP server and skills, and is designed to ensure that work generated by coding agents is visual and inspectable. It respects the permissions of the Dataiku API key used to connect, supports migration of legacy workflows from SAS, Alteryx and Excel, and includes administrative tools for finding failing jobs."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1793134194425423450"
              ],
              "manifest_id": "1793133898238389637"
            },
            {
              "claim_ids": [
                "1793124875173846746"
              ],
              "manifest_id": "1793124531092636500"
            },
            {
              "claim_ids": [
                "1793112082029321878"
              ],
              "manifest_id": "1793111719500630125"
            }
          ],
          "markdown": "Put together, the week describes a maturing craft: harder public benchmarks, private evaluation for custom mods, cheaper decision models for structured steps, and output that a human can inspect and govern. Teams that adopt all four habits will ship coding agents they can defend; teams that adopt only the model will keep discovering its limits in production."
        }
      ]
    }
  ],
  "unresolved": [
    "Whether higher reasoning effort can be made to help reliably on FrontierCode-style tasks",
    "How teams should evaluate mods when no reference judge exists for their domain",
    "Whether Jev-style decision models generalize beyond TypeSafe workflow evals"
  ],
  "decision_card": {
    "not_for": [
      "Teams looking for a single model score to settle a purchase",
      "Anyone deploying unsandboxed agent mods without review"
    ],
    "useful_for": [
      "Developers shipping coding agents or custom agent mods",
      "Teams choosing benchmarks and evaluation harnesses for code agents"
    ],
    "compare_against": [
      "Single-model benchmark leaderboards",
      "Unmeasured agent customization"
    ]
  },
  "writer": {
    "agent": "muse",
    "run_id": "2026-10-28T00"
  },
  "corrections": [],
  "cited_manifest_ids": [
    "1793133898238389637",
    "1793124531092636500",
    "1793128461355145256",
    "1793111719500630125"
  ],
  "manifests": [
    {
      "title": "FrontierCode 2.0",
      "source_url": "https://cognition.com/blog/frontier-code-2.0",
      "manifest_id": "1793133898238389637",
      "source_name": "windsurf-blog",
      "published_date": "2026-10-27"
    },
    {
      "title": "I put a ghost in Claude Code. Then I evaluated it",
      "source_url": "https://arize.com/blog/i-put-a-ghost-in-claude-code-then-i-evaluated-it",
      "manifest_id": "1793124531092636500",
      "source_name": "arize-ai-blog",
      "published_date": "2026-10-27"
    },
    {
      "title": "Building Faster AI Agents with Jev Decision Models and Cline",
      "source_url": "https://cline.ghost.io/jev-decision-models-ai-agents-cline",
      "manifest_id": "1793128461355145256",
      "source_name": "cline-blog",
      "published_date": "2026-10-27"
    },
    {
      "title": "Dataiku Headless: build in your favorite coding agent, review and ship in Dataiku",
      "source_url": "https://blog.dataiku.com/blog/dataiku-headless",
      "manifest_id": "1793111719500630125",
      "source_name": "dataiku-blog",
      "published_date": "2026-10-27"
    }
  ],
  "run_these_manifests": {
    "title": "Run these Manifests yourself",
    "summary": "Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.",
    "manifest_ids": [
      "1793133898238389637",
      "1793124531092636500",
      "1793128461355145256",
      "1793111719500630125"
    ],
    "steps": [
      {
        "n": 1,
        "title": "Get a Synorb Key",
        "text": "Click Generate Free Agent Key at https://synorb.com/connect/start, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{\"client\":\"your-agent-name\"}`. The key is the `api_key` field of the answer. Save the key. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons."
      },
      {
        "n": 2,
        "title": "Read each cited Manifest by id",
        "text": "REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief."
      },
      {
        "n": 3,
        "title": "Check the claim",
        "text": "Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them."
      }
    ],
    "mcp_url": "https://mcp.synorb.com/mcp",
    "connect_url": "https://synorb.com/connect/start",
    "request": {
      "method": "GET",
      "url_template": "https://api.synorb.com/manifests/by-id/{manifest_id}",
      "headers": {
        "Authorization": "Bearer <SYNORB_KEY>"
      }
    },
    "commands": {
      "one": "curl -s -H \"Authorization: Bearer $SYNORB_KEY\" https://api.synorb.com/manifests/by-id/1793133898238389637",
      "all": "for id in 1793133898238389637 1793124531092636500 1793128461355145256 1793111719500630125; do\n  curl -s -H \"Authorization: Bearer $SYNORB_KEY\" \"https://api.synorb.com/manifests/by-id/$id\"\ndone"
    }
  },
  "links": {
    "html": "https://hangingcontext.com/stories/2026-10-28/coding-agents-measured-not-shipped/",
    "json": "https://hangingcontext.com/stories/2026-10-28/coding-agents-measured-not-shipped/index.json",
    "markdown": "https://hangingcontext.com/stories/2026-10-28/coding-agents-measured-not-shipped/index.md",
    "newsroom": "https://hangingcontext.com/n/artificial-intelligence/",
    "edition": "https://hangingcontext.com/editions/2026-10-28T00/"
  },
  "related": [
    {
      "url": "https://hangingcontext.com/stories/2026-10-28/enterprise-agents-measured-work/",
      "headline": "Enterprise Agents Stop Demoing and Start Doing Measured Work",
      "slug": "enterprise-agents-measured-work"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-10-28/harness-geometry-agent-design/",
      "headline": "The Harness Is the Product: Why Agent Design Starts Outside the Model",
      "slug": "harness-geometry-agent-design"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-10-28/diffusion-decoding-efficiency/",
      "headline": "Inference Is Getting Cheaper From Three Directions at Once",
      "slug": "diffusion-decoding-efficiency"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-10-28/cross-embodiment-world-models/",
      "headline": "One World Model, Many Robot Bodies: The New Playbook for Robot Learning",
      "slug": "cross-embodiment-world-models"
    }
  ],
  "sponsor": {
    "label": "From Synorb",
    "title": "Ask what Synorb covers",
    "text": "The MCP tool synorb-stream-search answers questions like \"do you have this publisher?\" without using any Manifest allowance.",
    "cta": {
      "label": "Connect and ask",
      "url": "https://synorb.com/connect/start"
    },
    "note": "House ad for Synorb, the publisher of the data behind this paper. Not a news item."
  },
  "disclosure": "Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.",
  "reuse": {
    "version": "1",
    "summary": "Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product.",
    "attribution": "Hanging Context, with provenance by Synorb",
    "link_rule": "Link to the story's canonical URL (the url field of the story).",
    "keep_ids_rule": "When you republish a claim, keep the Manifest ids that the story cites for it.",
    "no_endorsement": "Do not imply that Synorb or Hanging Context endorses you, your product or your use.",
    "license_url": "https://hangingcontext.com/terms/reuse/",
    "machine_url": "https://hangingcontext.com/.well-known/reuse.json",
    "canonical_url": "https://hangingcontext.com/stories/2026-10-28/coding-agents-measured-not-shipped/"
  }
}