{
  "schema": "hangingcontext.desk/1",
  "type": "story",
  "id": "645",
  "slug": "security-agents-scan-code",
  "edition": "2026-10-27T00",
  "url": "https://hangingcontext.com/stories/2026-10-27/security-agents-scan-code/",
  "newsroom": "artificial-intelligence",
  "beat": "developers",
  "newsroom_title": "Artificial Intelligence",
  "angle": "CodeRabbit and Devin put frontier models on security review, while Fiddler warns frontier evaluators cost far more than small ones.",
  "headline": "Security Review Is Becoming an Agent Job, With a Human Still Signing Off",
  "dek": "CodeRabbit now uses GPT-6.1 Sol as the primary model in its security agent, Cognition upgraded Devin Security with frontier-model deep scans, and Fiddler AI's research on LLM judges argues for small in-environment evaluators over frontier models for routine checks.",
  "published_at": "2026-10-27T00:55:25Z",
  "labels": {
    "trust": "confirmed",
    "trust_text": "Confirmed · 3 sources",
    "newsroom": "Artificial Intelligence"
  },
  "source_count": 3,
  "publishers": [
    "fiddler-ai-publishing",
    "openai-news",
    "windsurf-blog"
  ],
  "key_facts": [
    {
      "text": "CodeRabbit uses GPT-6.1 Sol as the primary model in its security agent, and the system can attempt exploit verification in an isolated sandbox, the company says.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1793055159062744803",
          "claim_ids": [
            "1793055397087168983",
            "1793055397100423198"
          ]
        }
      ]
    },
    {
      "text": "Devin Security now offers deep scans powered by Claude Mythos and other frontier cyber models, Cognition says, and can write, validate, and test large-scale fixes on real VMs.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1793038512712313206",
          "claim_ids": [
            "1793038867413254642",
            "1793038867417595686"
          ]
        }
      ]
    },
    {
      "text": "Frontier model judges cost approximately two orders of magnitude more per evaluation than small models, according to Fiddler AI's research.",
      "self_reported": false,
      "cites": [
        {
          "manifest_id": "1793049058294591288",
          "claim_ids": [
            "1793049344169052553"
          ]
        }
      ]
    },
    {
      "text": "Deterministic checks should be used for any evaluation task with an exact answer, according to Fiddler AI.",
      "self_reported": false,
      "cites": [
        {
          "manifest_id": "1793049058294591288",
          "claim_ids": [
            "1793049344164211532"
          ]
        }
      ]
    }
  ],
  "images": [],
  "sections": [
    {
      "heading": "Review beyond the pull request",
      "paragraphs": [
        {
          "markdown": "CodeRabbit uses GPT-6.1 Sol as the primary model in its security agent, the company says, and GPT-6.1 Sol helps CodeRabbit extend security review beyond individual code changes. The system connects behavior across files and components to surface security issues, and Deep Scan supports broader investigation across repositories, including dependencies and component interactions.",
          "cites": [
            {
              "manifest_id": "1793055159062744803",
              "claim_ids": [
                "1793055397087168983",
                "1793055397089151685",
                "1793055397092994512",
                "1793055397097410504"
              ]
            }
          ]
        },
        {
          "markdown": "That cross-file scope is the change from earlier review bots. Pull requests can affect shared libraries and dependencies elsewhere in an application, and CodeRabbit Secure extends investigation to linked repositories to help teams understand system-wide weaknesses, the company says. CodeRabbit Secure can attempt exploit verification in an isolated sandbox, and can draft a pull request with a proposed fix for supported issues.",
          "cites": [
            {
              "manifest_id": "1793055159062744803",
              "claim_ids": [
                "1793055397091946390",
                "1793055397098425627",
                "1793055397100423198",
                "1793055397102972339"
              ]
            }
          ]
        },
        {
          "markdown": "Two limits are stated plainly. CodeRabbit's system complements established security scanners rather than replacing them, the company says, and human reviewers remain responsible for deciding which changes to ship. The agent proposes and verifies; the ship decision stays with a person.",
          "cites": [
            {
              "manifest_id": "1793055159062744803",
              "claim_ids": [
                "1793055397095699441",
                "1793055397103767598"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "Deep scans with a threat model",
      "paragraphs": [
        {
          "markdown": "Cognition is launching a major upgrade to Devin Security for finding, prioritizing, and remediating vulnerabilities, the company says. Devin Security now offers deep scans powered by Claude Mythos and other frontier cyber models, and it is model-neutral, able to use various cyber-capable models across labs.",
          "cites": [
            {
              "manifest_id": "1793038512712313206",
              "claim_ids": [
                "1793038867412956506",
                "1793038867413254642",
                "1793038867420228402"
              ]
            }
          ]
        },
        {
          "markdown": "Devin Security assesses exploitability and potential damage based on a custom threat model built with the client team, the company says, rather than applying a generic severity ranking. It can write, validate, and test large-scale fixes on real virtual machines running real services in parallel, and it can apply mitigations such as WAF rules, changes to security groups, or removal of public access to a service.",
          "cites": [
            {
              "manifest_id": "1793038512712313206",
              "claim_ids": [
                "1793038867415504147",
                "1793038867417595686",
                "1793038867422870912"
              ]
            }
          ]
        },
        {
          "markdown": "Cognition is committing hundreds of millions of dollars in Devin credits to help enterprises find and fix critical vulnerabilities, the company says, and Devin Security can ingest findings via API and integrations from existing scanners, bug bounties, and advisories. The credits lower the trial cost; they do not change the need to validate fixes before deployment.",
          "cites": [
            {
              "manifest_id": "1793038512712313206",
              "claim_ids": [
                "1793038867418854583",
                "1793038867421372973"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "The evaluation bill",
      "paragraphs": [
        {
          "markdown": "Fiddler AI's research on LLM-as-a-judge evaluation adds a cost discipline that both launches need. Frontier model judges cost approximately two orders of magnitude more per evaluation than small models, according to the research, and absolute scores in single-output scoring are prone to drift when the judge model changes.",
          "cites": [
            {
              "manifest_id": "1793049058294591288",
              "claim_ids": [
                "1793049344169052553",
                "1793049344175227506"
              ]
            }
          ]
        },
        {
          "markdown": "The recommendation is to split the work. Deterministic checks should be used for any evaluation task with an exact answer, according to Fiddler, and production traffic should go to small in-environment evaluators while frontier models are reserved for sampled audits. Fiddler Centor Models perform evaluation in-environment without external API calls, the company says.",
          "cites": [
            {
              "manifest_id": "1793049058294591288",
              "claim_ids": [
                "1793049344164211532",
                "1793049344182291509"
              ]
            }
          ]
        },
        {
          "markdown": "The research also warns that LLM evaluators exhibit self-preference, favoring their own generated text, and that pairwise comparisons should be performed in both orders to mitigate position bias. A security agent that grades its own fixes with the same model that wrote them needs an independent check, or the pass rate will flatter the tool.",
          "cites": [
            {
              "manifest_id": "1793049058294591288",
              "claim_ids": [
                "1793049344180068974",
                "1793049344168038936"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "What to measure before trusting the agent",
      "paragraphs": [
        {
          "markdown": "Teams adopting these tools should measure three things on their own code: the false-positive rate on clean code, the share of findings that survive human review, and the time from finding to merged fix. Neither announcement publishes those rates, so the vendor claims describe capability, not measured accuracy on a buyer's codebase.",
          "cites": [
            {
              "manifest_id": "1793055159062744803",
              "claim_ids": [
                "1793055397103767598"
              ]
            }
          ]
        },
        {
          "markdown": "The evaluation discipline from Fiddler applies directly to security agents: use deterministic checks where an answer is exact, reserve frontier models for sampled audits, and run pairwise comparisons in both orders to mitigate position bias. A security program that cannot say how its own judges are evaluated is, in effect, grading itself.",
          "cites": [
            {
              "manifest_id": "1793049058294591288",
              "claim_ids": [
                "1793049344164211532",
                "1793049344168038936"
              ]
            }
          ]
        }
      ]
    }
  ],
  "unresolved": [
    "Optimal threshold for switching between small and frontier models for specific tasks",
    "Measured false-positive and false-negative rates for the new deep scans",
    "Whether sandbox exploit verification covers the full set of supported issue types"
  ],
  "decision_card": {
    "useful_for": [
      "Adding agent security review alongside existing scanners"
    ],
    "not_for": [
      "Replacing established scanners or human review with agent output"
    ],
    "compare_against": [
      "Fiddler AI's judge research and vendor scan documentation"
    ]
  },
  "writer": {
    "agent": "muse",
    "run_id": "2026-10-27T00"
  },
  "corrections": [],
  "cited_manifest_ids": [
    "1793055159062744803",
    "1793038512712313206",
    "1793049058294591288"
  ],
  "manifests": [
    {
      "manifest_id": "1793055159062744803",
      "title": "GPT‑6.1 Sol helps CodeRabbit strengthen security across complex codebases",
      "source_name": "openai-news",
      "source_url": "https://openai.com/index/gpt-6-1-sol-coderabbit-code-security",
      "published_date": "2026-10-26"
    },
    {
      "manifest_id": "1793038512712313206",
      "title": "Devin Security: Find and fix your most dangerous vulnerabilities",
      "source_name": "windsurf-blog",
      "source_url": "https://cognition.com/blog/devin-security",
      "published_date": "2026-10-26"
    },
    {
      "manifest_id": "1793049058294591288",
      "title": "LLM-as-a-Judge: How It Works, Where It Fails, and What It Costs in Production",
      "source_name": "fiddler-ai-publishing",
      "source_url": "https://fiddler.ai/blog/llm-as-a-judge-production-costs-biases",
      "published_date": "2026-10-26"
    }
  ],
  "run_these_manifests": {
    "title": "Run these Manifests yourself",
    "summary": "Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.",
    "manifest_ids": [
      "1793055159062744803",
      "1793038512712313206",
      "1793049058294591288"
    ],
    "steps": [
      {
        "n": 1,
        "title": "Get a Synorb Key",
        "text": "Click Generate Free Agent Key at https://synorb.com/connect/start, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{\"client\":\"your-agent-name\"}`. The key is the `api_key` field of the answer. Save the key. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons."
      },
      {
        "n": 2,
        "title": "Read each cited Manifest by id",
        "text": "REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief."
      },
      {
        "n": 3,
        "title": "Check the claim",
        "text": "Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them."
      }
    ],
    "mcp_url": "https://mcp.synorb.com/mcp",
    "connect_url": "https://synorb.com/connect/start",
    "request": {
      "method": "GET",
      "url_template": "https://api.synorb.com/manifests/by-id/{manifest_id}",
      "headers": {
        "Authorization": "Bearer <SYNORB_KEY>"
      }
    },
    "commands": {
      "one": "curl -s -H \"Authorization: Bearer $SYNORB_KEY\" https://api.synorb.com/manifests/by-id/1793055159062744803",
      "all": "for id in 1793055159062744803 1793038512712313206 1793049058294591288; do\n  curl -s -H \"Authorization: Bearer $SYNORB_KEY\" \"https://api.synorb.com/manifests/by-id/$id\"\ndone"
    }
  },
  "links": {
    "html": "https://hangingcontext.com/stories/2026-10-27/security-agents-scan-code/",
    "json": "https://hangingcontext.com/stories/2026-10-27/security-agents-scan-code/index.json",
    "markdown": "https://hangingcontext.com/stories/2026-10-27/security-agents-scan-code/index.md",
    "newsroom": "https://hangingcontext.com/n/artificial-intelligence/",
    "edition": "https://hangingcontext.com/editions/2026-10-27T00/"
  },
  "related": [
    {
      "url": "https://hangingcontext.com/stories/2026-10-27/robotics-vla-data-engines/",
      "headline": "Robot Brains Are Getting Better Data, Better Views, and a Sense of Done",
      "slug": "robotics-vla-data-engines"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-10-27/anthropic-ipo-filing-builders/",
      "headline": "Anthropic Files for an IPO, and Its Filing Reads Like a Builder Risk Register",
      "slug": "anthropic-ipo-filing-builders"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-10-27/agents-learn-without-human-data/",
      "headline": "Self-Training, Team Training, and the Collaboration Gap",
      "slug": "agents-learn-without-human-data"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-10-26/multi-agent-teams-need-rails/",
      "headline": "Helpful agents are bypassing your workflows, and structure is the fix",
      "slug": "multi-agent-teams-need-rails"
    }
  ],
  "sponsor": {
    "label": "From Synorb",
    "title": "Claims with a source link",
    "text": "A Manifest holds a Signal of structured claims and a Brief, plus the source it came from. Your agent can check a sentence against it.",
    "cta": {
      "label": "Read the guide",
      "url": "https://synorb.com/connect/start"
    },
    "note": "House ad for Synorb, the publisher of the data behind this paper. Not a news item."
  },
  "disclosure": "Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.",
  "reuse": {
    "version": "1",
    "summary": "Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product.",
    "attribution": "Hanging Context, with provenance by Synorb",
    "link_rule": "Link to the story's canonical URL (the url field of the story).",
    "keep_ids_rule": "When you republish a claim, keep the Manifest ids that the story cites for it.",
    "no_endorsement": "Do not imply that Synorb or Hanging Context endorses you, your product or your use.",
    "license_url": "https://hangingcontext.com/terms/reuse/",
    "machine_url": "https://hangingcontext.com/.well-known/reuse.json",
    "canonical_url": "https://hangingcontext.com/stories/2026-10-27/security-agents-scan-code/"
  }
}