Agents can't check their own work: the verifiers they generate reject one in three right answers

The verifiers are too strict

In a preprint this week, VerifierBench takes on a problem every agent team has quietly hit: you ask an LLM agent to generate a verifier — code that checks whether a task was done right — and the verifier turns out to be the unreliable part. Across 19 models tested in a common harness, normalized balanced accuracy on verifier generation ranged from 10.6 to 62.1. Nobody is good at this yet. m:…6640

The failure mode is specific and costly. Generated verifiers consistently reject invalid solutions more reliably than they accept valid ones — they are strict in the wrong direction. Most false rejections in the analyzed records came from verifiers that accepted no tested solution to a task at all, and GPT-6-astra rejected 33% of correct answers in baseline testing. Case studies trace this to unsupported assumptions the verifiers make about solution formats, tool use and code changes. m:…6640

The practical sting: if you use agent-generated verifiers to filter synthetic training data or gate task completion, you are silently throwing away good work. The paper's proposed fix, the Gavel harness with its evidence-based rule system, reached a score of 68.3 with GPT-6-astra — better, but still a long way from trustworthy. The benchmark also ships a V4V system that performs a six-check audit on source verifiers and test suites — a verifier for the verifiers, since the generation step itself can't be trusted. m:…6640

Errors hide in the traces

A second preprint looks one level up the stack, at autonomous deep search and deep research agents — the systems that plan, browse and synthesize over long runs. The authors' diagnostic framework finds these agents are highly susceptible to errors in intermediate execution traces that final-outcome evaluation never captures. The answer can look right while the reasoning that produced it went wrong somewhere in the middle. The danger is a model that reaches the right answer through broken reasoning and gets full marks for it. m:…0996

To make the failures legible, the paper proposes a structured taxonomy dividing agent vulnerabilities into system-level and task-level errors. For developers shipping research agents, the takeaway is that end-to-end benchmarks are insufficient: you need to instrument and audit the intermediate steps, because that is where the errors live. m:…0996

Even the judges need testing

A third preprint closes the loop: if you use LLM judges to evaluate any of the above, the judges themselves need a testing harness. The SANE framework automates scorer-specific metamorphic testing for LLM judges, generating 3.7 times as many validated test families as the existing Judge Reliability Harness. m:…9521

The alignment numbers are stark. SANE reached a Spearman rho of 0.852 in judge-ranking alignment with human-labeled FaithBench, while the baseline managed 0.100 — essentially no better than chance at ranking judges the way humans do. SANE also detected scorer deficiencies around professionalism and toxicity that the baseline missed entirely. m:…9521

Put together, the three papers describe an evaluation stack that is broken at every layer: agents generate bad verifiers, research agents hide errors in their traces, and the judges that score everything need their own tests. For developers, the honest response is defense in depth — audit the verifier, trace the intermediate steps, and test the judge — rather than trusting any single check. m:…6640 m:…0996 m:…9521