Security Review Is Becoming an Agent Job, With a Human Still Signing Off

Review beyond the pull request

CodeRabbit uses GPT-6.1 Sol as the primary model in its security agent, the company says, and GPT-6.1 Sol helps CodeRabbit extend security review beyond individual code changes. The system connects behavior across files and components to surface security issues, and Deep Scan supports broader investigation across repositories, including dependencies and component interactions. m:…4803

That cross-file scope is the change from earlier review bots. Pull requests can affect shared libraries and dependencies elsewhere in an application, and CodeRabbit Secure extends investigation to linked repositories to help teams understand system-wide weaknesses, the company says. CodeRabbit Secure can attempt exploit verification in an isolated sandbox, and can draft a pull request with a proposed fix for supported issues. m:…4803

Two limits are stated plainly. CodeRabbit's system complements established security scanners rather than replacing them, the company says, and human reviewers remain responsible for deciding which changes to ship. The agent proposes and verifies; the ship decision stays with a person. m:…4803

Deep scans with a threat model

Cognition is launching a major upgrade to Devin Security for finding, prioritizing, and remediating vulnerabilities, the company says. Devin Security now offers deep scans powered by Claude Mythos and other frontier cyber models, and it is model-neutral, able to use various cyber-capable models across labs. m:…3206

Devin Security assesses exploitability and potential damage based on a custom threat model built with the client team, the company says, rather than applying a generic severity ranking. It can write, validate, and test large-scale fixes on real virtual machines running real services in parallel, and it can apply mitigations such as WAF rules, changes to security groups, or removal of public access to a service. m:…3206

Cognition is committing hundreds of millions of dollars in Devin credits to help enterprises find and fix critical vulnerabilities, the company says, and Devin Security can ingest findings via API and integrations from existing scanners, bug bounties, and advisories. The credits lower the trial cost; they do not change the need to validate fixes before deployment. m:…3206

The evaluation bill

Fiddler AI's research on LLM-as-a-judge evaluation adds a cost discipline that both launches need. Frontier model judges cost approximately two orders of magnitude more per evaluation than small models, according to the research, and absolute scores in single-output scoring are prone to drift when the judge model changes. m:…1288

The recommendation is to split the work. Deterministic checks should be used for any evaluation task with an exact answer, according to Fiddler, and production traffic should go to small in-environment evaluators while frontier models are reserved for sampled audits. Fiddler Centor Models perform evaluation in-environment without external API calls, the company says. m:…1288

The research also warns that LLM evaluators exhibit self-preference, favoring their own generated text, and that pairwise comparisons should be performed in both orders to mitigate position bias. A security agent that grades its own fixes with the same model that wrote them needs an independent check, or the pass rate will flatter the tool. m:…1288

What to measure before trusting the agent

Teams adopting these tools should measure three things on their own code: the false-positive rate on clean code, the share of findings that survive human review, and the time from finding to merged fix. Neither announcement publishes those rates, so the vendor claims describe capability, not measured accuracy on a buyer's codebase. m:…4803

The evaluation discipline from Fiddler applies directly to security agents: use deterministic checks where an answer is exact, reserve frontier models for sampled audits, and run pairwise comparisons in both orders to mitigate position bias. A security program that cannot say how its own judges are evaluated is, in effect, grading itself. m:…1288