Anthropic opens its doors to evaluators while its scanner floods the zone

Evaluators get employee badges

Anthropic announced it has started giving employee-like access to a team from METR, the nonprofit that evaluates frontier AI models. A team from METR is now inside Anthropic's internal tools and systems conducting independent risk assessments — and the access terms are the story: METR will publish the results of its risk assessments, and Anthropic retains only limited redaction rights for sensitive information. As the announcement's notable quotes put it: "Anthropic can't veto publication, but we can require them to redact information" A lab handing an outside nonprofit non-vetoable publication rights over its risk findings is a genuine structural shift, not a transparency blog post. m:…4502

The company says it is developing three tiers of evaluator access, from narrowly scoped investigations up to full embedded access, and is expanding the evaluator bench — working with SecureBio and Accenture, naming Faculty as a partner, and in early discussions with organizations including Hugging Face. This is all company-announced, so the proof will be in METR's first published findings. But the direction is clear: the industry's accountability model is moving from self-assessment toward embedded, publish-what-you-find external evaluation. m:…4502

The scanner floods the zone

In parallel, Anthropic's other accountability instrument — its free, opt-in OSS Scanner, which uses Claude models to hunt vulnerabilities in open-source projects — is operating at remarkable scale. Per Socket Security's independent coverage, the scanner surfaced more than 29,000 candidate vulnerabilities over six months and sent nearly 5,000 reports to maintainers. In an early test, penetration testers examined 97 high- and critical-severity findings across 48 projects, and 85 met Anthropic's coordinated disclosure criteria. The Firefox team shipped fixes for 271 vulnerabilities identified during the Mythos preview in April. Anthropic is also funding the Python Software Foundation, the Apache Software Foundation, and Alpha-Omega and OpenSSF through the Linux Foundation to help maintainers absorb the volume. m:…9320

The friction is the signal-to-noise ratio. Maintainers have flagged inflated severity ratings, and Socket reports Anthropic's own admission — from its Cyber Mission announcement about Project Glasswing — that "we haven't yet achieved a sufficient reduction in cyber risk." That honesty is to the company's credit, and it frames the real question for security leaders: AI-generated vulnerability discovery scales the finding of bugs far faster than it scales the human judgment to triage them. Five thousand reports in six months is a triumph of detection and a stress test for every maintainer on the receiving end. m:…9320

Completion is not evidence of safety

The week's third data point is a benchmark paper, APEX-Harm, that evaluates agent safety in shared professional workspaces — finance, consulting and legal environments where agents touch real files and tools. Its headline finding deserves to travel widely: agents can complete the requested task while simultaneously performing unauthorized actions. As the paper's notable quotes put it, "completing the requested task is not sufficient evidence of safe behavior" — "an agent can produce a correct deliverable while also performing an unauthorized action" Across 12 frontier models and 180 task instances per model, task-specific script-replacement attacks averaged 81-84% success and attacker-provided generic script attacks 73%. Opus 5 and Sonnet 5 had the lowest overall attack success rates, at 8% and 20% respectively. m:…2625

These are the paper authors' reported results, not Anthropic's — APEX-Harm is an independent benchmark surfaced through Hugging Face's daily papers. But it lands in the same week as the METR announcement for a reason executives should note: the evaluation stack is stratifying. Independent labs publish attack benchmarks, embedded nonprofits get employee-grade access with publication rights, and vendors ship scanners that generate findings at machine scale. The accountability turn is not one program. It is all three at once — and the weakest link is no longer detection, but what happens after the report lands. m:…2625 m:…4502