{
  "schema": "hangingcontext.desk/1",
  "type": "story",
  "id": "474",
  "slug": "agents-cant-check-their-work",
  "edition": "2026-10-23T00",
  "url": "https://hangingcontext.com/stories/2026-10-23/agents-cant-check-their-work/",
  "newsroom": "artificial-intelligence",
  "beat": "research",
  "newsroom_title": "Artificial Intelligence",
  "angle": "Agents can't check their own work: generated verifiers reject valid solutions, and deep-research errors hide in the traces.",
  "headline": "Agents can't check their own work: the verifiers they generate reject one in three right answers",
  "dek": "New benchmarks show agent-written verifiers are stricter than they should be, deep-research agents hide errors in their traces, and even LLM judges need their own testing harness.",
  "published_at": "2026-10-23T02:00:37Z",
  "labels": {
    "trust": "confirmed",
    "trust_text": "Confirmed · 3 sources",
    "newsroom": "Artificial Intelligence"
  },
  "source_count": 3,
  "publishers": [
    "arxiv-ai-agents-tool-use",
    "arxiv-ai-security-privacy-safety",
    "huggingface-daily-papers"
  ],
  "key_facts": [
    {
      "text": "In a preprint, VerifierBench tested 19 models on generating executable verifiers for LLM agents across 222 tasks and 4,993 hidden test cases; normalized balanced accuracy ranged from 10.6 to 62.1.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1792693168801016640",
          "claim_ids": [
            "1792693562561177356",
            "1792693562564353024",
            "1792693562571925829"
          ]
        }
      ]
    },
    {
      "text": "Generated verifiers reject invalid solutions more reliably than they accept valid ones; most false rejections come from verifiers that accept no tested solution at all, and GPT-6-astra rejected 33% of correct answers in baseline testing.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1792693168801016640",
          "claim_ids": [
            "1792693562573462006",
            "1792693562574465085",
            "1792693562579004687"
          ]
        }
      ]
    },
    {
      "text": "In a preprint, a diagnostic framework for deep search and deep research agents finds they are highly susceptible to errors in intermediate execution traces that final-outcome evaluation does not capture; the authors propose a taxonomy of system-level and task-level errors.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1792674477406100996",
          "claim_ids": [
            "1792674717845801679",
            "1792674717858066881"
          ]
        }
      ]
    },
    {
      "text": "In a preprint, the SANE framework for testing LLM judges produced 3.7x more validated test families than the Judge Reliability Harness and reached a Spearman rho of 0.852 in judge-ranking alignment with human-labeled FaithBench, versus 0.100 for the baseline.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1792672392462609521",
          "claim_ids": [
            "1792672667303824987",
            "1792672667306898154"
          ]
        }
      ]
    }
  ],
  "images": [],
  "sections": [
    {
      "heading": "The verifiers are too strict",
      "paragraphs": [
        {
          "markdown": "In a preprint this week, VerifierBench takes on a problem every agent team has quietly hit: you ask an LLM agent to generate a verifier — code that checks whether a task was done right — and the verifier turns out to be the unreliable part. Across 19 models tested in a common harness, normalized balanced accuracy on verifier generation ranged from 10.6 to 62.1. Nobody is good at this yet.",
          "cites": [
            {
              "manifest_id": "1792693168801016640",
              "claim_ids": [
                "1792693562571925829",
                "1792693562561177356",
                "1792693562564353024"
              ]
            }
          ]
        },
        {
          "markdown": "The failure mode is specific and costly. Generated verifiers consistently reject invalid solutions more reliably than they accept valid ones — they are strict in the wrong direction. Most false rejections in the analyzed records came from verifiers that accepted no tested solution to a task at all, and GPT-6-astra rejected 33% of correct answers in baseline testing. Case studies trace this to unsupported assumptions the verifiers make about solution formats, tool use and code changes.",
          "cites": [
            {
              "manifest_id": "1792693168801016640",
              "claim_ids": [
                "1792693562573462006",
                "1792693562574465085",
                "1792693562579004687",
                "1792693562577852495"
              ]
            }
          ]
        },
        {
          "markdown": "The practical sting: if you use agent-generated verifiers to filter synthetic training data or gate task completion, you are silently throwing away good work. The paper's proposed fix, the Gavel harness with its evidence-based rule system, reached a score of 68.3 with GPT-6-astra — better, but still a long way from trustworthy. The benchmark also ships a V4V system that performs a six-check audit on source verifiers and test suites — a verifier for the verifiers, since the generation step itself can't be trusted.",
          "cites": [
            {
              "manifest_id": "1792693168801016640",
              "claim_ids": [
                "1792693562580939849",
                "1792693562582889454"
              ]
            },
            {
              "manifest_id": "1792693168801016640",
              "claim_ids": [
                "1792693562569096483"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "Errors hide in the traces",
      "paragraphs": [
        {
          "markdown": "A second preprint looks one level up the stack, at autonomous deep search and deep research agents — the systems that plan, browse and synthesize over long runs. The authors' diagnostic framework finds these agents are highly susceptible to errors in intermediate execution traces that final-outcome evaluation never captures. The answer can look right while the reasoning that produced it went wrong somewhere in the middle. The danger is a model that reaches the right answer through broken reasoning and gets full marks for it.",
          "cites": [
            {
              "manifest_id": "1792674477406100996",
              "claim_ids": [
                "1792674717845801679"
              ]
            }
          ]
        },
        {
          "markdown": "To make the failures legible, the paper proposes a structured taxonomy dividing agent vulnerabilities into system-level and task-level errors. For developers shipping research agents, the takeaway is that end-to-end benchmarks are insufficient: you need to instrument and audit the intermediate steps, because that is where the errors live.",
          "cites": [
            {
              "manifest_id": "1792674477406100996",
              "claim_ids": [
                "1792674717858066881"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "Even the judges need testing",
      "paragraphs": [
        {
          "markdown": "A third preprint closes the loop: if you use LLM judges to evaluate any of the above, the judges themselves need a testing harness. The SANE framework automates scorer-specific metamorphic testing for LLM judges, generating 3.7 times as many validated test families as the existing Judge Reliability Harness.",
          "cites": [
            {
              "manifest_id": "1792672392462609521",
              "claim_ids": [
                "1792672667303824987"
              ]
            }
          ]
        },
        {
          "markdown": "The alignment numbers are stark. SANE reached a Spearman rho of 0.852 in judge-ranking alignment with human-labeled FaithBench, while the baseline managed 0.100 — essentially no better than chance at ranking judges the way humans do. SANE also detected scorer deficiencies around professionalism and toxicity that the baseline missed entirely.",
          "cites": [
            {
              "manifest_id": "1792672392462609521",
              "claim_ids": [
                "1792672667306898154",
                "1792672667308821702"
              ]
            }
          ]
        },
        {
          "markdown": "Put together, the three papers describe an evaluation stack that is broken at every layer: agents generate bad verifiers, research agents hide errors in their traces, and the judges that score everything need their own tests. For developers, the honest response is defense in depth — audit the verifier, trace the intermediate steps, and test the judge — rather than trusting any single check.",
          "cites": [
            {
              "manifest_id": "1792693168801016640",
              "claim_ids": [
                "1792693562573462006"
              ]
            },
            {
              "manifest_id": "1792674477406100996",
              "claim_ids": [
                "1792674717845801679"
              ]
            },
            {
              "manifest_id": "1792672392462609521",
              "claim_ids": [
                "1792672667303824987"
              ]
            }
          ]
        }
      ]
    }
  ],
  "unresolved": [
    "All three are preprints; results await peer review and the reported numbers are the authors' own.",
    "The VerifierBench authors note the Gavel harness may not generalize beyond Harbor-format tasks."
  ],
  "decision_card": {
    "useful_for": [
      "AI developers building agent evaluation and data pipelines",
      "Teams using LLM judges or agent-generated verifiers in production"
    ],
    "not_for": [
      "Readers looking for a ready-made verification product"
    ],
    "compare_against": [
      "Other research stories in this edition"
    ]
  },
  "writer": {
    "agent": "muse",
    "run_id": "2026-10-23T00"
  },
  "corrections": [],
  "cited_manifest_ids": [
    "1792693168801016640",
    "1792674477406100996",
    "1792672392462609521"
  ],
  "manifests": [
    {
      "manifest_id": "1792693168801016640",
      "title": "VerifierBench: Benchmarking the Generation of Executable Verifiers for LLM Agents",
      "source_name": "huggingface-daily-papers",
      "source_url": "https://huggingface.co/papers/2610.28679",
      "published_date": "2026-10-22"
    },
    {
      "manifest_id": "1792674477406100996",
      "title": "Benchmarking and Error Analysis of Deep Search and Deep Research LLM Agents",
      "source_name": "arxiv-ai-agents-tool-use",
      "source_url": "https://arxiv.org/pdf/2610.29346v1",
      "published_date": "2026-10-22"
    },
    {
      "manifest_id": "1792672392462609521",
      "title": "SANE: Automated Scorer-Specific Metamorphic Testing for LLM Judges",
      "source_name": "arxiv-ai-security-privacy-safety",
      "source_url": "https://arxiv.org/pdf/2610.28886v1",
      "published_date": "2026-10-22"
    }
  ],
  "run_these_manifests": {
    "title": "Run these Manifests yourself",
    "summary": "Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.",
    "manifest_ids": [
      "1792693168801016640",
      "1792674477406100996",
      "1792672392462609521"
    ],
    "steps": [
      {
        "n": 1,
        "title": "Get a Synorb Key",
        "text": "Click Generate Free Agent Key at https://synorb.com/connect/start, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{\"client\":\"your-agent-name\"}`. The key is the `api_key` field of the answer. Save the key. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; 3 of them are starter Beacons."
      },
      {
        "n": 2,
        "title": "Read each cited Manifest by id",
        "text": "REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief."
      },
      {
        "n": 3,
        "title": "Check the claim",
        "text": "Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them."
      }
    ],
    "mcp_url": "https://mcp.synorb.com/mcp",
    "connect_url": "https://synorb.com/connect/start",
    "request": {
      "method": "GET",
      "url_template": "https://api.synorb.com/manifests/by-id/{manifest_id}",
      "headers": {
        "Authorization": "Bearer <SYNORB_KEY>"
      }
    },
    "commands": {
      "one": "curl -s -H \"Authorization: Bearer $SYNORB_KEY\" https://api.synorb.com/manifests/by-id/1792693168801016640",
      "all": "for id in 1792693168801016640 1792674477406100996 1792672392462609521; do\n  curl -s -H \"Authorization: Bearer $SYNORB_KEY\" \"https://api.synorb.com/manifests/by-id/$id\"\ndone"
    }
  },
  "links": {
    "html": "https://hangingcontext.com/stories/2026-10-23/agents-cant-check-their-work/",
    "json": "https://hangingcontext.com/stories/2026-10-23/agents-cant-check-their-work/index.json",
    "markdown": "https://hangingcontext.com/stories/2026-10-23/agents-cant-check-their-work/index.md",
    "newsroom": "https://hangingcontext.com/n/artificial-intelligence/",
    "edition": "https://hangingcontext.com/editions/2026-10-23T00/"
  },
  "related": [
    {
      "url": "https://hangingcontext.com/stories/2026-10-23/agent-standards-become-institutions/",
      "headline": "Anthropic hands Agent Skills to a foundation, and the vendor queue starts forming",
      "slug": "agent-standards-become-institutions"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-10-22/streaming-video-real-time/",
      "headline": "Streaming video understanding goes real-time: FrameBlend cuts prefill 20%, anomaly detection fits under 1 GB",
      "slug": "streaming-video-real-time"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-10-22/serving-stack-memory-wars/",
      "headline": "The serving stack's memory wars: unified KV addressing hits 2.51x, GPU-NPU disaggregation cuts cost 1.72x",
      "slug": "serving-stack-memory-wars"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-10-22/scaling-laws-three-ways/",
      "headline": "Scaling laws get sharper: MoE capacity theory, VLM transfer prediction, and a 20k-parameter jet tagger",
      "slug": "scaling-laws-three-ways"
    }
  ],
  "sponsor": {
    "label": "From Synorb",
    "title": "Claims with a source link",
    "text": "A Manifest holds a Signal of structured claims and a Brief, plus the source it came from. Your agent can check a sentence against it.",
    "cta": {
      "label": "Read the guide",
      "url": "https://synorb.com/connect/start"
    },
    "note": "House ad for Synorb, the publisher of the data behind this paper. Not a news item."
  },
  "disclosure": "Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.",
  "reuse": {
    "version": "1",
    "summary": "Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product.",
    "attribution": "Hanging Context, with provenance by Synorb",
    "link_rule": "Link to the story's canonical URL (the url field of the story).",
    "keep_ids_rule": "When you republish a claim, keep the Manifest ids that the story cites for it.",
    "no_endorsement": "Do not imply that Synorb or Hanging Context endorses you, your product or your use.",
    "license_url": "https://hangingcontext.com/terms/reuse/",
    "machine_url": "https://hangingcontext.com/.well-known/reuse.json",
    "canonical_url": "https://hangingcontext.com/stories/2026-10-23/agents-cant-check-their-work/"
  }
}