GPT-6 Cyber aces the scripted attacks and stalls on hardened targets
Two independent evaluations of GPT-6 Cyber agree on the shape of the result: it finds more, finishes scripted attacks, and still fails against hardened targets.
- CodeRabbit found GPT-6 Cyber caught 22 selected issues versus 19 without it, with 66% more high- and medium-severity findings — but says it still needs to check how many of the extra findings are real vulnerabilities. m:…9136
- Irregular Research found GPT-6 Cyber solved all 11 CyScenarioBench scenarios and all 22 Atomic challenges at least once, and 90 of 288 FrontierCyber challenges — with no successful attacks on fully hardened targets. m:…3802
- GPT-6 Astra, the general model, solved 96 of 288 FrontierCyber challenges versus 90 for the security variant, and neither solved any of the 15 highest-difficulty challenges. m:…3802
- Merge, which builds on OpenAI technology, reports tool calls through its own systems succeed 99.9% of the time, versus around 60% for public MCP servers. Self-reported m:…5285
What the evaluators found
CodeRabbit evaluated GPT-6 Cyber, a cybersecurity variant of OpenAI's GPT-6 models trained for security research, inside its CodeRabbit Secure platform, with access provided through the OpenAI Daybreak Defense Network. The evaluation covered Python, JavaScript, TypeScript, Go, PHP, Rust, Ruby, C, and C#. The ensemble with GPT-6 Cyber caught 22 selected issues compared with 19 without it, and reported 66% more high- and medium-severity findings. Injection and server-side request forgery showed the largest percentage increases, and denial of service added 98 findings while authorization bypass added 77. m:…9136
Irregular Research, which collaborated with OpenAI on its evaluation, tested the model across three offensive security benchmarks: FrontierCyber, CyScenarioBench, and Atomic Challenges. GPT-6 Cyber solved all 11 CyScenarioBench scenarios at least once and all 22 Atomic challenges at least once, and 90 of 288 FrontierCyber challenges. FrontierCyber tests vulnerability discovery and exploitation against real software and hardware, and the evaluators observed no successful attacks on fully hardened targets. On Atomic Challenges, the model reached a 97% success rate in vulnerability research and exploitation, 98% in network attack simulation, and 63% in evasion. m:…3802
The comparison with the general model is the part that should slow down any simple story about specialization. GPT-6 Astra solved 96 of 288 FrontierCyber challenges, more than the security variant's 90, and neither model solved any of the 15 highest-difficulty FrontierCyber challenges. Where GPT-6 Cyber did pull ahead was on multi-stage work: it completed 11 of 11 CyScenarioBench scenarios against 8 of 11 for Astra, with an average success rate of 61%, up 7 percentage points. The pattern is that the security tuning helps most on long, staged operations, not on raw exploitation. m:…3802
The false-positive problem nobody has solved yet
CodeRabbit is candid about the gap in its own result: it needs to assess how many of the additional findings are real vulnerabilities. A 66% rise in high- and medium-severity findings is only good news if the extra findings are true positives. Between October 23 and November 3, teams ran more than 13,000 Deep Scans across more than 2,700 organizations, so the volume of findings at stake is large — and a model that cries wolf more often will be tuned out by the developers it is supposed to help. Any team adopting GPT-6 Cyber for review should measure precision, not just recall, before trusting the count. m:…9136
Merge, a startup that builds product and business tools with OpenAI technology, offers a useful counterpoint from the defensive side of the same week. It uses GPT-6 Astra, ChatGPT, and Codex across product reviews, marketing, and business development planning, and reports that tool calls through its own systems succeed 99.9% of the time, compared with around 60% for public MCP servers. Its Merge Agent Handler provides secure tool access for agents, and its Gateway provides model routing and cost controls. The implication is that the reliability gap in agent tooling is currently a plumbing problem as much as a model problem — and the plumbing is where a builder can actually control the outcome. m:…5285
What it means for developers
For a developer deciding whether to put a security-tuned model into a review pipeline, the week's evidence supports a narrow conclusion: GPT-6 Cyber is good at finding known classes of weakness and completing staged attack scenarios, and it is not a skeleton key. CyScenarioBench measures multi-stage offensive operations over long horizons, and Atomic Challenges measure focused skills in vulnerability research, network attack simulation, and evasion — exactly the skills that matter for an attacker working through a real system, and exactly where the model still hits a ceiling on hardened targets. m:…3802
The practical move is to treat the model as a first pass with a human in the loop, and to track how many of its findings survive review. Merge's chief executive Shensi Ding advises executives to experiment with AI in low-risk areas before scaling, which is the right posture here: run the security model on non-critical repositories first, measure the true-positive rate, and only then let it near the code that matters. A model that finds 66% more issues is only an improvement if the extra issues are real. m:…9136 m:…5285
What is still unresolved
- How many of CodeRabbit's additional GPT-6 Cyber findings are real vulnerabilities rather than false positives.
- Whether GPT-6 Cyber's advantage on multi-stage operations holds against targets harder than the current benchmarks.
- What the Daybreak Defense Network's access terms mean for who can evaluate future security models independently.
Provenance by Synorb. Every claim links to its source. This story was written by an agent from the Manifests listed below. Read the methodology and limitations, or send a correction.
Free to reuse, including commercially. Credit “Hanging Context, with provenance by Synorb” and link to this story. Keep the cited Manifest ids. Reuse terms v1.