{
  "schema": "hangingcontext.desk/1",
  "type": "story",
  "id": "1718",
  "slug": "verified-fixes-not-findings",
  "edition": "2026-11-18T00",
  "url": "https://hangingcontext.com/stories/2026-11-18/verified-fixes-not-findings/",
  "newsroom": "artificial-intelligence",
  "beat": "developers",
  "newsroom_title": "Artificial Intelligence",
  "angle": "AI finds bugs fast and cheap. The bottleneck moved to proving the fix, and three posts show the verification layer being built.",
  "headline": "AI finds the bug in seconds. Proving the fix is the new bottleneck",
  "dek": "A Microsoft pilot shipped verified fixes, a migration test needed an independent verifier, and enterprise buyers want one testing platform. Verification is where agent coding now stalls.",
  "published_at": "2026-11-18T00:51:27Z",
  "labels": {
    "trust": "confirmed",
    "trust_text": "Confirmed · 3 Manifests",
    "newsroom": "Artificial Intelligence"
  },
  "source_count": 3,
  "publishers": [
    "github-old-blog",
    "snowflake-medium-rss",
    "uipath-blog"
  ],
  "sources": [
    "github-old-blog",
    "snowflake-medium-rss",
    "uipath-blog"
  ],
  "distinct_source_count": 3,
  "key_facts": [
    {
      "text": "The Codename MDASH pilot delivered 99 private, pre-verified vulnerability reports with suggested fixes; by November 15, 2026, maintainers had fixed 55 of them.",
      "cites": [
        {
          "claim_ids": [
            "1794956796167623631",
            "1794956796168623632"
          ],
          "manifest_id": "1794956581380602423"
        }
      ],
      "self_reported": true
    },
    {
      "text": "Maintainers of Caddy fixed all 22 reports they received within a week or less.",
      "cites": [
        {
          "claim_ids": [
            "1794956796173623638"
          ],
          "manifest_id": "1794956581380602423"
        }
      ],
      "self_reported": true
    },
    {
      "text": "A database-migration agent factory reached a 92% first-try pass rate with Claude 3.5 Sonnet translating and Claude 3.5 Opus verifying, because LLM judges rate their own output higher.",
      "cites": [
        {
          "claim_ids": [
            "1794944243621470922",
            "1794944243594470917"
          ],
          "manifest_id": "1794943996930468648"
        }
      ],
      "self_reported": true
    },
    {
      "text": "Enterprise buyers prefer unified testing platforms over tool sprawl, and AI testing shifts from deterministic verification to probabilistic trust evaluation.",
      "cites": [
        {
          "claim_ids": [
            "1794944837011479323",
            "1794944837010479321"
          ],
          "manifest_id": "1794944556734477076"
        }
      ],
      "self_reported": true
    }
  ],
  "images": [],
  "sections": [
    {
      "heading": "The pilot that shipped fixes",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1794956796166623630",
                "1794956796167623631",
                "1794956796170623634",
                "1794956796169623633",
                "1794956796168623632"
              ],
              "manifest_id": "1794956581380602423"
            }
          ],
          "markdown": "The pilot treated the bug report as the wrong unit of work. Codename MDASH, a multi-model agentic scanning harness developed by Microsoft, produced 99 private vulnerability reports for maintainers, and the point of the exercise was that each report arrived pre-verified, with a suggested fix attached. The post argues maintainers should only receive security reports that are pre-verified and include a suggested fix. Seven researchers from Microsoft FORGE Lab spent approximately two days per project manually verifying findings, and as of November 15, 2026, maintainers had fixed 55 of the 99 reports."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1794956796173623638",
                "1794956796171623635",
                "1794956796172623636",
                "1794956796172623637",
                "1794956796164623629"
              ],
              "manifest_id": "1794956581380602423"
            }
          ],
          "markdown": "The fastest maintainers needed no convincing. Maintainers of Caddy fixed all 22 reports they received within a week or less. Microsoft has since announced MAI Cyber 2, an updated version of the model used in the pilot, and GitHub will offer AI security scans to all alumni projects of the GitHub Secure Open Source Fund, which has supported 216 projects and over 300 maintainers. The argument running through the post is blunt: AI has made finding potential vulnerabilities fast and cheap, so the scarce thing is no longer the finding. It is the verified fix."
        }
      ]
    },
    {
      "heading": "The verifier has to be independent",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1794944243570470914",
                "1794944243626470923",
                "1794944243577470915",
                "1794944243637470925"
              ],
              "manifest_id": "1794943996930468648"
            }
          ],
          "markdown": "The same lesson shows up in a very different setting: migrating database logic. Steiner built a proof-of-concept agent factory for migrating PostgreSQL logic to Snowflake, using parallel AI agents for translation and scripted checks for validation, and the test that mattered was not whether the data matched. Data-matching is not equivalent to correct logic in database migrations, and the factory's design reflects that: agents are restricted from accessing databases directly, and only gate scripts perform database operations."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1794944243621470922",
                "1794944243594470917",
                "1794944243604470919",
                "1794944243654470929",
                "1794944243659470930"
              ],
              "manifest_id": "1794943996930468648"
            }
          ],
          "markdown": "The numbers are honest about the limits. The author achieved a 92% first-try pass rate using Claude 3.5 Sonnet as the translator and Claude 3.5 Opus as the verifier, and the verifier had to be a different model, because LLM judges rate their own output higher than equally good output from others. The author's own test harness caused more failures than the AI agents did, and a diagnoser was added to classify failures and reduce wasted retries. Snowflake announced Snowflake AIM in October 2026 for code conversion and validation, which suggests the vendors see the same gap."
        }
      ]
    },
    {
      "heading": "Testing is the bottleneck now",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1794944837007479318",
                "1794944837009479320",
                "1794944837010479321",
                "1794944837011479323",
                "1794944837012479324",
                "1794944837011479322",
                "1794944837013479325"
              ],
              "manifest_id": "1794944556734477076"
            }
          ],
          "markdown": "UiPath's blog makes the same point from the buyer's side. AI-driven coding tools are accelerating software development, and the bottleneck has moved to the testing phase. Testing must now encompass both traditional software and AI components, including LLMs and agents, and the shift is from deterministic verification to probabilistic trust evaluation. Enterprise buyers consistently prefer unified testing platforms over best-of-breed tool sprawl, and autonomous testing requires a control plane and visibility to ensure human oversight. Testers are shifting into strategic roles focused on quality standards and managing autonomous testing agents, and UiPath Test Cloud includes an AI Trust Layer for auditing AI activity and controlling data usage."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1794956796164623629"
              ],
              "manifest_id": "1794956581380602423"
            },
            {
              "claim_ids": [
                "1794944243577470915"
              ],
              "manifest_id": "1794943996930468648"
            },
            {
              "claim_ids": [
                "1794944837007479318"
              ],
              "manifest_id": "1794944556734477076"
            }
          ],
          "markdown": "The through-line is simple: finding things is solved, and proving things is not. The pilot that shipped fixes, the migration that needed an independent verifier, and the buyers who want one testing platform are all paying for the same layer. Teams that budget for agents should budget for the check, because the check is where the work now sits."
        }
      ]
    }
  ],
  "unresolved": [
    "The material does not say what happened to the reports that were not fixed by November 15.",
    "The migration test is one author's proof of concept, not a production rollout.",
    "UiPath's account of buyer preference is the company's own; no independent survey is cited."
  ],
  "decision_card": {
    "not_for": [
      "teams that only need a list of possible bugs"
    ],
    "useful_for": [
      "teams shipping agent-written code",
      "maintainers triaging AI security reports"
    ],
    "compare_against": [
      "raw scanner output with no verification step",
      "manual code review queues"
    ]
  },
  "writer": {
    "agent": "muse",
    "run_id": "2026-11-18T00"
  },
  "corrections": [],
  "cited_manifest_ids": [
    "1794956581380602423",
    "1794943996930468648",
    "1794944556734477076"
  ],
  "manifests": [
    {
      "title": "Maintainers don't need more AI bug reports. They need fixes.",
      "source_url": "https://github.blog/open-source/maintainers/maintainers-dont-need-more-ai-bug-reports-they-need-fixes",
      "manifest_id": "1794956581380602423",
      "source_name": "github-old-blog",
      "published_date": "2026-11-17"
    },
    {
      "title": "Building an Agent Factory for Data Migrations: a Small Test with Snowflake",
      "source_url": "https://medium.com/snowflake/building-an-agent-factory-for-data-migrations-a-small-test-with-snowflake-9d4622c8f2c7",
      "manifest_id": "1794943996930468648",
      "source_name": "snowflake-medium-rss",
      "published_date": "2026-11-17"
    },
    {
      "title": "Testing at the speed of AI: what our guest, a Forrester analyst, says enterprise buyers need next",
      "source_url": "https://uipath.com/blog/ai/ai-speed-testing-what-forrester-analyst-says-enterprise-buyers-need-next",
      "manifest_id": "1794944556734477076",
      "source_name": "uipath-blog",
      "published_date": "2026-11-17"
    }
  ],
  "run_these_manifests": {
    "title": "Run these Manifests yourself",
    "summary": "Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.",
    "manifest_ids": [
      "1794956581380602423",
      "1794943996930468648",
      "1794944556734477076"
    ],
    "steps": [
      {
        "n": 1,
        "title": "Get a Synorb Key",
        "text": "Click Generate Free Agent Key at https://synorb.com/connect/start, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{\"client\":\"your-agent-name\"}`. The key is the `api_key` field of the answer. Save the key. Existing 72 Hour Agent Demo keys keep their issued limits. New keys require email OTP verification. An existing demo key lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons."
      },
      {
        "n": 2,
        "title": "Read each cited Manifest by id",
        "text": "REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief."
      },
      {
        "n": 3,
        "title": "Check the claim",
        "text": "Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them."
      }
    ],
    "mcp_url": "https://mcp.synorb.com/mcp",
    "connect_url": "https://synorb.com/connect/start",
    "request": {
      "method": "GET",
      "url_template": "https://api.synorb.com/manifests/by-id/{manifest_id}",
      "headers": {
        "Authorization": "Bearer <SYNORB_KEY>"
      }
    },
    "commands": {
      "one": "curl -s -H \"Authorization: Bearer $SYNORB_KEY\" https://api.synorb.com/manifests/by-id/1794956581380602423",
      "all": "for id in 1794956581380602423 1794943996930468648 1794944556734477076; do\n  curl -s -H \"Authorization: Bearer $SYNORB_KEY\" \"https://api.synorb.com/manifests/by-id/$id\"\ndone"
    }
  },
  "links": {
    "html": "https://hangingcontext.com/stories/2026-11-18/verified-fixes-not-findings/",
    "json": "https://hangingcontext.com/stories/2026-11-18/verified-fixes-not-findings/index.json",
    "markdown": "https://hangingcontext.com/stories/2026-11-18/verified-fixes-not-findings/index.md",
    "newsroom": "https://hangingcontext.com/n/artificial-intelligence/",
    "edition": "https://hangingcontext.com/editions/2026-11-18T00/"
  },
  "related": [
    {
      "url": "https://hangingcontext.com/stories/2026-11-22/where-your-prompts-actually-go/",
      "headline": "Three inference providers disclosed prompt failures in the same week",
      "slug": "where-your-prompts-actually-go"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-11-22/the-weeks-plumbing-papers/",
      "headline": "Three preprints fix the plumbing: which data to keep, which causes matter, when robots must stop",
      "slug": "the-weeks-plumbing-papers"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-11-22/the-agent-platform-standardizes/",
      "headline": "The agent platform is standardising: protocols, observability, governance",
      "slug": "the-agent-platform-standardizes"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-11-22/patch-the-ai-stack/",
      "headline": "Four patches, one lesson: the AI stack's weak points are the boring parts",
      "slug": "patch-the-ai-stack"
    }
  ],
  "sponsor": {
    "label": "From Synorb",
    "title": "Ask what Synorb covers",
    "text": "The MCP tool synorb-stream-search answers questions like \"do you have this publisher?\" without using any Manifest allowance.",
    "cta": {
      "label": "Connect and ask",
      "url": "https://synorb.com/connect/start"
    },
    "note": "House ad for Synorb, the publisher of the data behind this paper. Not a news item."
  },
  "disclosure": "Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.",
  "reuse": {
    "version": "1",
    "summary": "Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product.",
    "attribution": "Hanging Context, with provenance by Synorb",
    "link_rule": "Link to the story's canonical URL (the url field of the story).",
    "keep_ids_rule": "When you republish a claim, keep the Manifest ids that the story cites for it.",
    "no_endorsement": "Do not imply that Synorb or Hanging Context endorses you, your product or your use.",
    "license_url": "https://hangingcontext.com/terms/reuse/",
    "machine_url": "https://hangingcontext.com/.well-known/reuse.json",
    "canonical_url": "https://hangingcontext.com/stories/2026-11-18/verified-fixes-not-findings/"
  }
}
