Coding agents get measured: FrontierCode 2.0 tops out at 45 percent while teams learn to evaluate their own mods

The benchmark gets harder

The Cognition Team released FrontierCode 2.0 as an updated coding benchmark with stricter mergeability standards and broader task coverage, and the headline result is humbling: Claude Opus 5.5 is the top-performing model with a score of 45 percent. Agentic software engineering remains far from solved, the team says, and the number gives teams a realistic ceiling for what current models do on hard, real-world-style tasks. m:…9637

The benchmark is harder in measurable ways. FrontierCode 2.0 tasks are drawn from 55 percent more repositories than FrontierCode 1.0, thoroughness criteria grew by 3x compared with FrontierCode 1.1, and tasks include 43 percent more blocking criteria than previous versions. In 80 percent of tasks, the reference solution differs from the upstream pull request, up from 60 percent in FrontierCode 1.0, which means models cannot coast by reproducing a known patch. m:…9637

Models struggle to infer user intent, respect unstated constraints and anticipate long-term consequences of their changes, according to the benchmark authors, and models can score lower at higher reasoning effort levels on FrontierCode 2.0. That second finding is worth pausing on: more thinking does not automatically mean better code, and teams that pay for higher effort should check whether the extra tokens buy accuracy on their own repositories. m:…9637

One measure did improve: the failure rate for scope criteria in FrontierCode 2.0 is 3.8 percent, compared with 9.0 percent in FrontierCode 1.1. Scope discipline is getting better even as overall tasks get harder, which suggests models are learning to stay inside the task, even when they cannot yet finish it well. m:…9637

Your own mods need measuring too

Benchmarks for frontier models are only half the story, because teams now customize their coding agents. Arize describes Claude Code mods as small TypeScript functions that allow users to rewrite prompts, add UI or replace built-in features, and notes that Claude Code currently ships six built-in mods, five of which are enabled by default. One of them, the agents-md mod, was developed in response to user requests for Claude Code to read AGENTS.md files. m:…6500

The caution in the Arize post is direct: Claude Code mods are not sandboxed, creating potential privacy or security risks, and a regex-based guard in a mod does not function as a secure sandbox. A ghost mod built to evaluate coding turns adds a median of 1.7 seconds to each turn, a small cost, but tracing agent sessions with Arize AX provides visibility into mod costs and execution without determining whether mod judgments are accurate. m:…6500

Evaluating agent mods requires a reference judge and a versioned dataset to distinguish between mod bugs and policy disagreements, Arize argues, and LLM judges must be provided with the same context as the agent being evaluated to avoid false negatives. A larger model does not necessarily perform better as a judge if it lacks the same input context as the agent. The lesson generalizes: a team that ships a custom agent modification without an evaluation harness is flying blind, however good the base model is. m:…6500

Faster decisions, inspectable output

TypeSafe AI introduced Jev, a decision model designed to address latency and cost challenges in traditional LLMs, and Cline has integrated it through plugins for browser navigation, prompt testing and tool-call guardrails. Jev classifies input states into choices, scores or probabilities instead of emitting tokens, and Jev-guard allows instant guardrails to judge tool calls before LLM execution. TypeSafe says Jev is approximately 194 times faster and 445 times cheaper than traditional LLMs on its workflow evals, figures that are company-reported and should be tested on a team's own decision tasks. m:…5256

Cline also offers ClinePass, a $9.99 per month subscription plan providing access to open-weight models like DeepSeek, Kimi, GLM and Qwen, alongside plugins Jev-browser, which provides a real browser inside Cline for automated navigation, and Jev-playground, which translates vague prompts into Jev requests via a web interface. The combination points to a division of labor: small decision models for structured choices, large models for language, and a subscription tier for open-weight experimentation. m:…5256

Dataiku Headless closes the loop on the governance side, allowing developers to use coding agents to build and modify Dataiku projects, with Cobuild generating data pipelines, models and dashboards. It is available as a plugin for Claude Code, Codex, Cursor, Snowflake CoCo, AWS Kiro and OpenCode, can be connected using an MCP server and skills, and is designed to ensure that work generated by coding agents is visual and inspectable. It respects the permissions of the Dataiku API key used to connect, supports migration of legacy workflows from SAS, Alteryx and Excel, and includes administrative tools for finding failing jobs. m:…0125

Put together, the week describes a maturing craft: harder public benchmarks, private evaluation for custom mods, cheaper decision models for structured steps, and output that a human can inspect and govern. Teams that adopt all four habits will ship coding agents they can defend; teams that adopt only the model will keep discovering its limits in production. m:…9637 m:…6500 m:…0125