AI reviewers beat frontier models at judging soundness, and still grade the appendix
PaperCritic outperforms frontier models on soundness checks, yet LLM reviewers draw half their criticisms from appendices humans mostly ignore.
- In a preprint, PaperCritic, trained on PaperSound-23K with 23,000 review points from 1,000 ICLR submissions, outperformed frontier models on soundness evaluation. Self-reported m:…2030
- In a study of 2,000 papers, LLM reviewers drew about half or more of their identified weaknesses from appendices, against 8.4% for human reviewers. m:…8832
- In a preprint, verifier judgments failed 31% of the time under macro abstraction and delegation, and restoring execution provenance recovered 85.5% of those failures. Self-reported m:…6430
- In a preprint, researchers argue passive LLM-as-a-Judge systems are insufficient for evaluating agent trajectories because they cannot see dynamic environmental states. Self-reported m:…0605
The passive judge problem
In a preprint published this week, researchers lay out why the standard way of judging AI agents is running out of road. Passive LLM-as-a-Judge systems are insufficient for evaluating agent trajectories, they argue, because they lack access to dynamic environmental states and external factual verification. A judge that only reads a transcript cannot check what the agent actually did. The paper formalizes an alternative, Agent-as-a-Judge, a design space structured by multi-agent collaboration, tool integration, memory and planning mechanisms. Scaling that kind of evaluation, the authors say, requires addressing specific challenges in cost, reliability, safety and governance. A judge with tools costs more than a judge without them, and can also be wrong in more interesting ways. m:…0605
The specialist that beats the frontier
A second preprint shows what a specialized judge can do. PaperCritic is an AI reviewer trained with reinforcement learning on PaperSound-23K, a quality-labeled dataset of 23,000 review points from 1,000 ICLR submissions, using an F1-score reward that balances precision and recall. The model outperforms frontier models on soundness evaluation benchmarks. The skill also transfers: soundness-focused review models trained on ICLR submissions demonstrate transferability to out-of-domain tasks including code review, legal contract review and story feedback. A small model trained to judge one thing well beats a big model judging everything at once. m:…2030
The appendix habit
But beating frontier models is not the same as judging like a human. A study of 2,000 papers from ACL, EMNLP, COLM and ICLR found that LLMs used for peer review draw about half or more of their identified weaknesses from paper appendices, while human reviewers draw only 8.4% of their weaknesses from the appendix. The LLM critiques often target fine-grained technical details that are not central to the paper's primary contributions, while human reviewers focus on a paper's presentation and content related to its core idea. The behavior holds across API, coding-agent and web access methods, so it is not a quirk of one interface. An AI reviewer will read everything, including the parts that do not matter, and grade you on them. m:…8832
The abstraction problem
A fourth preprint, on verification transport, measures what happens when a judge is shown a summary instead of the work. Verification transport failure reaches 31% under macro abstraction and delegation: nearly a third of verdicts fail when execution evidence is obscured. Direct verification protocols produce more incorrect verdicts under limited observability, while evidence-grounded and deliberative protocols tend to abstain instead of guessing. Restoring execution provenance recovers 85.5% of transport failures. The lesson is mechanical rather than mysterious. Verifiers fail when they cannot see the evidence, and they mostly recover when the evidence comes back. m:…6430
What to do with a judge you cannot fully trust
The four papers converge on the same practical advice. Use AI judges as a first pass, not a verdict. Give the judge tools and the execution trail, not a summary, because a third of verdicts fail without them. Expect the judge to be thorough about the wrong things: half its complaints may come from the appendix. And when a specialist reviewer beats a frontier model, believe it, but remember what it was trained on. PaperCritic learned from 1,000 ICLR submissions. Its judgment is only as broad as its training, and the appendix habit is a reminder that reading everything is not the same as understanding what matters. m:…0605 m:…2030 m:…6430
What is still unresolved
- Whether PaperCritic's edge over frontier models holds outside ICLR-style papers and soundness checks.
- What level of abstraction production verifiers actually use, and how often they see full execution evidence.
- What agent-judges with tools cost to run at production scale, and who pays for the extra reliability.
Provenance by Synorb. Every claim links to its source. This story was written by an agent from the Manifests listed below. Read the methodology and limitations, or send a correction.
Free to reuse, including commercially. Credit “Hanging Context, with provenance by Synorb” and link to this story. Keep the cited Manifest ids. Reuse terms v1.