--- title: The smallest Claude out-engineered the biggest GPT — judgment, not size, decided it description: Surge AI's 60-task benchmark found Claude models ahead on all eight dimensions, while a one-line cache setting cut Claude Code bills 10-15%. canonical: https://hangingcontext.com/stories/2026-11-14/smallest-claude-better-engineer-than-biggest-gpt/ last-updated: 2026-11-14 url: https://hangingcontext.com/stories/2026-11-14/smallest-claude-better-engineer-than-biggest-gpt/ published: 2026-11-14T01:16:58Z edition: 2026-11-14T00 newsroom: artificial-intelligence beat: developers trust: confirmed source_count: 3 cited_manifest_ids: ['1794606227234416091', '1794601798040385316', '1794604607367405094'] --- # The smallest Claude out-engineered the biggest GPT — judgment, not size, decided it *Surge AI's 60-task benchmark found Claude models ahead on all eight dimensions, while a one-line cache setting cut Claude Code bills 10-15%.* Confirmed · 3 Manifests · Artificial Intelligence · Published 2026-11-14 01:16 UTC ## Key facts - Claude models outscored GPT models across all eight dimensions of Surge AI's sudo L7 software engineering benchmark, which tested five frontier models on 60 tasks. [cites: 1794606227234416091] - Claude Haiku 5.5 identified a security risk in a B2B codebase task that GPT-6.1 Sol missed, and GPT Sol failed to spot a configuration change that would cause infrastructure destruction. [cites: 1794606227234416091] - Larridin reduced Claude Code token costs by 10 to 15 percent by changing one configuration setting, after Anthropic cut Claude Fable 5.1 cache-read prices by 75 percent. (self-reported) [cites: 1794601798040385316] - In reviewer experiments described by Pydantic, Codex with gpt-6-luna scored 3.93, ahead of Cursor at 2.80 and Claude Code at 2.60. [cites: 1794604607367405094] ## What Surge AI found Surge AI evaluated five frontier models on 60 tasks from its sudo L7 software engineering benchmark, and Claude models outscored GPT models across all eight dimensions. The gap was not about writing code that runs. But Claude models were better at the parts of engineering that surround the code: identifying security risks, questioning flawed specifications, and communicating potential system consequences, according to Surge AI. The finding suggests that effective AI engineering agents need judgment beyond simple code implementation. [cites: 1794606227234416091] The examples make the point concrete. Claude Haiku 5.5, the smallest Claude in the test, identified a security risk in a B2B codebase task that GPT-6.1 Sol missed. GPT Sol also failed to identify that a requested configuration change would cause infrastructure destruction and service interruption. On another task, Claude models correctly identified that caching customer account information, including balances, would lead to stale data issues, while GPT models missed that risk. And Claude models write roughly ten times longer final reports than GPT models, surfacing risks and caveats that GPT models omit. [cites: 1794606227234416091] ## The price of judgment The cheapest model per task was not the cheapest to run. GPT Sol costs $0.91 per task, while Claude Haiku costs $1.68 per task, but Surge AI's argument is that the cost-efficiency of cheaper models is offset by the potential for expensive errors, such as production incidents or data exposure, which require manual review to mitigate. GPT models also produce roughly four times as many inaccurate statements per word as Claude models when normalized by response length. A model that is cheap per task and wrong in ways that need a human to catch is not cheap. [cites: 1794606227234416091] > **From Synorb** — Ask what Synorb covers. The MCP tool synorb-stream-search answers questions like "do you have this publisher?" without using any Manifest allowance. [Connect and ask](https://synorb.com/connect/start) There is also a cheaper way to run the better-judgment model. Larridin's CTO reduced Claude Code token costs by 10 to 15 percent by changing one configuration setting, after Anthropic reduced the price of Claude Fable 5.1 cache reads by 75 percent, to $0.25 per million tokens. The largest component of a Claude Code bill is the model re-reading existing context, and cache reads cost $0.25 per million tokens while cache misses cost $10 per million. Setting promptCacheTtl to one hour keeps the main conversation cache alive longer, and the setting requires Claude Code v2.1.242 or later. The default cache lifetime for API key usage is five minutes, which is why sessions paused for a break were paying to rebuild context. [cites: 1794601798040385316] ## Judgment is not one skill The honest complication is that writing code and reviewing it are different jobs, and the rankings flip. Pydantic's account of coding agent traces notes that review-fix and review phases account for approximately 60 percent of total software factory pipeline costs, which is why tracing matters: coding harnesses are often opaque, hiding model invocations and tool calls, and traces let teams diagnose failures, branch turns for apples-to-apples experiments, and let agents reuse past session context to reduce redundant exploration. In reviewer experiments described in the same piece, Codex with gpt-6-luna outperformed Cursor and Claude Code, scoring 3.93 against 2.80 and 2.60. The model that writes with the best judgment is not necessarily the one that reviews best. [cites: 1794604607367405094] For teams choosing a coding model, the practical read is to stop ranking models on one axis. Surge AI's benchmark says Claude is the safer pair of hands for writing and changing production code, because it questions the request and reports what could go wrong. The reviewer numbers say Codex deserves the review seat. And the caching story says the bill for either choice depends heavily on configuration, not just the model's sticker price. The teams that will get the most from these models are the ones that measure judgment, review quality and cost separately, and assign each job to the model that wins it. [cites: 1794606227234416091, 1794604607367405094, 1794601798040385316] ## What is still unresolved - Whether Surge AI's 60-task benchmark generalizes beyond the tasks it chose, and how its eight dimensions were weighted. - Whether Claude's longer reports actually get read by engineers, or just cost more tokens. - How the reviewer scores were produced, and whether they hold across different codebases. ## Decision card - **Useful for:** Engineering leads choosing coding models; Teams costing Claude Code deployments; Buyers of agent observability tooling - **Not for:** Anyone picking a model on benchmark scores alone; Teams that cannot review agent output - **Compare against:** GPT-6.1 Sol on cost per task; Codex as a reviewer; Cursor as a reviewer ## Provenance ledger | Manifest id | Title | Source | Claims used | Published | |---|---|---|---|---| | `1794606227234416091` | The Smallest Claude Is a Better Software Engineer Than the Biggest GPT | [surge-ai-blog](https://surgehq.ai/blog/claude-vs-gpt-software-engineering) | 1794606558242418713, 1794606558239418712, 1794606558244418714, 1794606558251418719, 1794606558245418715, 1794606558252418720, 1794606558254418721, 1794606558247418716, 1794606558249418718, 1794606558248418717 | 11-13 00:00 | | `1794601798040385316` | Claude Code Prompt Caching: Cut Token Costs 10-15% | [larridin-blog](https://larridin.com/blog/claude-code-prompt-caching) | 1794602001838386428, 1794602001834386427, 1794602001840386429, 1794602001841386430, 1794602001843386431, 1794602001844386432, 1794602001847386434 | 11-13 00:00 | | `1794604607367405094` | Using coding agent traces to improve your workflows | [pydantic-articles](https://pydantic.dev/articles/coding-harness-tracing) | 1794604926521408379, 1794604926526408382, 1794604926528408383, 1794604926530408385, 1794604926515408376, 1794604926532408386, 1794604926507408370, 1794604926511408373 | 11-13 00:00 | ## Run these Manifests yourself Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them. 1. **Get a Synorb Key.** Click Generate Free Agent Key at https://synorb.com/connect/start, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{"client":"your-agent-name"}`. The key is the `api_key` field of the answer. Save the key. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons. 2. **Read each cited Manifest by id.** REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer `. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief. 3. **Check the claim.** Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them. Cited Manifest ids: - `1794606227234416091` - `1794601798040385316` - `1794604607367405094` ```sh for id in 1794606227234416091 1794601798040385316 1794604607367405094; do curl -s -H "Authorization: Bearer $SYNORB_KEY" "https://api.synorb.com/manifests/by-id/$id" done ``` > **From Synorb** — Run the cited Manifests yourself. Every claim in this paper points to a Manifest id. Get a Synorb Key and read the Manifests behind it. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons. [Get a key](https://synorb.com/connect/start) ## Related stories - [The AI bill is now a line item, and four levers are cutting it without cutting capability](https://hangingcontext.com/stories/2026-11-14/cutting-ai-bill-four-levers/) - [AI writes the code now. Humans can't read it fast enough — and open source just proved it](https://hangingcontext.com/stories/2026-11-15/ai-prs-broke-open-source-review/) - [Agents claim work they never did. Now we can count how often](https://hangingcontext.com/stories/2026-11-15/agents-claim-work-they-never-did/) - [Throughput is the new model spec, and the gap between hosts is now fortyfold](https://hangingcontext.com/stories/2026-11-14/inference-throughput-race-kimi-deepseek/) ## Links - HTML: https://hangingcontext.com/stories/2026-11-14/smallest-claude-better-engineer-than-biggest-gpt/ - JSON: https://hangingcontext.com/stories/2026-11-14/smallest-claude-better-engineer-than-biggest-gpt/index.json - Markdown: https://hangingcontext.com/stories/2026-11-14/smallest-claude-better-engineer-than-biggest-gpt/index.md Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify. --- Reuse (terms v1): Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product. Credit: "Hanging Context, with provenance by Synorb", with a link to https://hangingcontext.com/stories/2026-11-14/smallest-claude-better-engineer-than-biggest-gpt/. Terms: https://hangingcontext.com/terms/reuse/