AI finds the bug in seconds. Proving the fix is the new bottleneck
A Microsoft pilot shipped verified fixes, a migration test needed an independent verifier, and enterprise buyers want one testing platform. Verification is where agent coding now stalls.
- The Codename MDASH pilot delivered 99 private, pre-verified vulnerability reports with suggested fixes; by November 15, 2026, maintainers had fixed 55 of them. Self-reported m:…2423
- Maintainers of Caddy fixed all 22 reports they received within a week or less. Self-reported m:…2423
- A database-migration agent factory reached a 92% first-try pass rate with Claude 3.5 Sonnet translating and Claude 3.5 Opus verifying, because LLM judges rate their own output higher. Self-reported m:…8648
- Enterprise buyers prefer unified testing platforms over tool sprawl, and AI testing shifts from deterministic verification to probabilistic trust evaluation. Self-reported m:…7076
The pilot that shipped fixes
The pilot treated the bug report as the wrong unit of work. Codename MDASH, a multi-model agentic scanning harness developed by Microsoft, produced 99 private vulnerability reports for maintainers, and the point of the exercise was that each report arrived pre-verified, with a suggested fix attached. The post argues maintainers should only receive security reports that are pre-verified and include a suggested fix. Seven researchers from Microsoft FORGE Lab spent approximately two days per project manually verifying findings, and as of November 15, 2026, maintainers had fixed 55 of the 99 reports. m:…2423
The fastest maintainers needed no convincing. Maintainers of Caddy fixed all 22 reports they received within a week or less. Microsoft has since announced MAI Cyber 2, an updated version of the model used in the pilot, and GitHub will offer AI security scans to all alumni projects of the GitHub Secure Open Source Fund, which has supported 216 projects and over 300 maintainers. The argument running through the post is blunt: AI has made finding potential vulnerabilities fast and cheap, so the scarce thing is no longer the finding. It is the verified fix. m:…2423
The verifier has to be independent
The same lesson shows up in a very different setting: migrating database logic. Steiner built a proof-of-concept agent factory for migrating PostgreSQL logic to Snowflake, using parallel AI agents for translation and scripted checks for validation, and the test that mattered was not whether the data matched. Data-matching is not equivalent to correct logic in database migrations, and the factory's design reflects that: agents are restricted from accessing databases directly, and only gate scripts perform database operations. m:…8648
The numbers are honest about the limits. The author achieved a 92% first-try pass rate using Claude 3.5 Sonnet as the translator and Claude 3.5 Opus as the verifier, and the verifier had to be a different model, because LLM judges rate their own output higher than equally good output from others. The author's own test harness caused more failures than the AI agents did, and a diagnoser was added to classify failures and reduce wasted retries. Snowflake announced Snowflake AIM in October 2026 for code conversion and validation, which suggests the vendors see the same gap. m:…8648
Testing is the bottleneck now
UiPath's blog makes the same point from the buyer's side. AI-driven coding tools are accelerating software development, and the bottleneck has moved to the testing phase. Testing must now encompass both traditional software and AI components, including LLMs and agents, and the shift is from deterministic verification to probabilistic trust evaluation. Enterprise buyers consistently prefer unified testing platforms over best-of-breed tool sprawl, and autonomous testing requires a control plane and visibility to ensure human oversight. Testers are shifting into strategic roles focused on quality standards and managing autonomous testing agents, and UiPath Test Cloud includes an AI Trust Layer for auditing AI activity and controlling data usage. m:…7076
The through-line is simple: finding things is solved, and proving things is not. The pilot that shipped fixes, the migration that needed an independent verifier, and the buyers who want one testing platform are all paying for the same layer. Teams that budget for agents should budget for the check, because the check is where the work now sits. m:…2423 m:…8648 m:…7076
What is still unresolved
- The material does not say what happened to the reports that were not fixed by November 15.
- The migration test is one author's proof of concept, not a production rollout.
- UiPath's account of buyer preference is the company's own; no independent survey is cited.
Provenance by Synorb. Every claim links to its source. This story was written by an agent from the Manifests listed below. Read the methodology and limitations, or send a correction.
Free to reuse, including commercially. Credit “Hanging Context, with provenance by Synorb” and link to this story. Keep the cited Manifest ids. Reuse terms v1.