{
  "schema": "hangingcontext.desk/1",
  "type": "story",
  "id": "1781",
  "slug": "benchmarks-audit-trust-not-scores",
  "edition": "2026-11-19T12",
  "url": "https://hangingcontext.com/stories/2026-11-19/benchmarks-audit-trust-not-scores/",
  "newsroom": "artificial-intelligence",
  "beat": "research",
  "newsroom_title": "Artificial Intelligence",
  "angle": "New benchmarks audit whether an AI can be trusted with money, politics and children, not just whether it gets the answer right.",
  "headline": "The new benchmarks do not ask if the model is smart. They ask if it can be trusted.",
  "dek": "In one new audit, 72.99% of trading points show at least one gap. In another, every model leans left-progressive on US content. A third finds expert-level science is not the same as winning tasks.",
  "published_at": "2026-11-19T12:39:41Z",
  "labels": {
    "trust": "reported",
    "trust_text": "Reported · 4 Manifests",
    "newsroom": "Artificial Intelligence"
  },
  "source_count": 4,
  "publishers": [
    "arxiv-llm-evals-benchmarks",
    "arxiv-rag-search-knowledge"
  ],
  "sources": [
    "arxiv-llm-evals-benchmarks",
    "arxiv-rag-search-knowledge"
  ],
  "distinct_source_count": 2,
  "key_facts": [
    {
      "text": "In a preprint, 72.99% of trading points in AStockArena exhibit at least one audit gap, and model-level audit gap rates range from 52.2% to 91.2%.",
      "cites": [
        {
          "claim_ids": [
            "1795078077901667778",
            "1795078077902667779"
          ],
          "manifest_id": "1795077718392666722"
        }
      ],
      "self_reported": true
    },
    {
      "text": "In a preprint, Polar: all evaluated models lean left-progressive on U.S. political content, while presentation language alone can shift measured political bias.",
      "cites": [
        {
          "claim_ids": [
            "1795076197418660388",
            "1795076197419660389",
            "1795076197422660390"
          ],
          "manifest_id": "1795075910574658127"
        }
      ],
      "self_reported": true
    },
    {
      "text": "In a preprint, LLMs exhibit systematic gender stereotyping in children's stories, with son-directed stories emphasizing agency and daughter-directed stories emphasizing communion.",
      "cites": [
        {
          "claim_ids": [
            "1795078077606667620"
          ],
          "manifest_id": "1795077718392666725"
        }
      ],
      "self_reported": true
    },
    {
      "text": "In a preprint, task-level wins and domain-level leadership are dissociated in frontier LLMs, and at least one model exceeded the expert anchor on 7 of 12 tasks in STEMBench.",
      "cites": [
        {
          "claim_ids": [
            "1795078076730667452",
            "1795078076734667457"
          ],
          "manifest_id": "1795077718392666726"
        }
      ],
      "self_reported": true
    }
  ],
  "images": [],
  "sections": [
    {
      "heading": "What changed",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1795078077901667778",
                "1795078077902667779",
                "1795078077903667780"
              ],
              "manifest_id": "1795077718392666722"
            }
          ],
          "markdown": "A financial agent can make money and still be untrustworthy, and a new preprint makes that distinction measurable. AStockArena is a decision-auditing benchmark for financial agents, and 72.99% of trading points in AStockArena exhibit at least one audit gap. Model-level audit gap rates range from 52.2% to 91.2%, so even the best model in the test leaves gaps in more than half of its trading decisions. Portfolio outcomes are sensitive to trades flagged by the AStockArena audit process, which means the flagged trades are not cosmetic errors. They are the trades that move the result, and the audit is catching the decisions where the agent's evidence or execution does not support what it did."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1795076197418660388",
                "1795076197419660389",
                "1795076197422660390"
              ],
              "manifest_id": "1795075910574658127"
            }
          ],
          "markdown": "A second preprint, Polar, applies the same audit instinct to politics. Polar is a benchmark, and political bias in LLMs varies systematically with political context, issue category, model group, and presentation language. All evaluated models lean left-progressive on U.S. political content, while showing more centered and mixed patterns on South Korean content. Presentation language alone can shift measured political bias in LLMs. That last finding is the one to sit with: the same model, asked about the same issue, can measure as more or less biased depending on the language of the question, which means a bias score without a language and a context attached is not a stable property of the model at all."
        }
      ]
    },
    {
      "heading": "Children's stories and expert science",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1795078077606667620",
                "1795078077608667622",
                "1795078077610667623"
              ],
              "manifest_id": "1795077718392666725"
            }
          ],
          "markdown": "A third preprint looks at where bias shows up when nobody is asking a political question. In a preprint on LLM-generated children's stories, the authors find that LLMs exhibit systematic gender stereotyping, with son-directed stories emphasizing agency and daughter-directed stories emphasizing communion. Hidden-state analysis reveals that LLMs contain decodable information regarding the recipient's gender, so the model is not just responding to the words in the prompt; it carries a representation of who the story is for. Narrator gender has limited moderation on gender-stereotyping patterns compared to the recipient's gender, which suggests the stereotyping is driven by who the story is about, not who is telling it. A story generator that quietly teaches different lessons to sons and daughters is a different kind of failure from a wrong answer, and it is harder to catch with a standard accuracy test."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1795078076727667450",
                "1795078076730667452",
                "1795078076734667457"
              ],
              "manifest_id": "1795077718392666726"
            }
          ],
          "markdown": "The fourth preprint, STEMBench, complicates the idea of expertise itself. STEMBench uses field-specific figures of merit and expert-measured anchors to evaluate scientific reasoning. Task-level wins and domain-level leadership are dissociated in frontier LLMs, so a model can win individual tasks without leading a field, and lead a field without winning every task. At least one model exceeded the expert anchor on 7 of 12 tasks, which sounds impressive until the dissociation is taken into account: beating an expert on a task is not the same as reasoning like an expert across a domain. The benchmark's point is that a single leaderboard number hides the shape of what a model can and cannot do."
        }
      ]
    },
    {
      "heading": "What this means",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1795078077901667778"
              ],
              "manifest_id": "1795077718392666722"
            },
            {
              "claim_ids": [
                "1795076197418660388"
              ],
              "manifest_id": "1795075910574658127"
            }
          ],
          "markdown": "The common thread across these four preprints is that accuracy is no longer the hard question. The hard question is whether the reasoning behind the answer holds up: whether a trading decision has evidence, whether a political answer is stable across language and context, whether a children's story treats its audience fairly, whether a scientific win reflects real understanding. Each benchmark is an attempt to make that question testable, and each one finds that current models fall short in a different way. For anyone deciding whether to deploy an agent in a regulated or high-stakes setting, the practical takeaway is to ask for the audit, not the score. A model that scores well and audits badly is a model that will produce a confident, well-formatted decision with a gap in the middle, and the gap is where the risk lives."
        }
      ]
    }
  ],
  "unresolved": [
    "Whether audit gaps in AStockArena predict real trading losses, or only mark decisions that are hard to explain.",
    "How much of Polar's language effect comes from translation versus cultural framing.",
    "Whether the stereotyping in children's stories changes when the model is given explicit guidance about the recipient."
  ],
  "decision_card": {
    "not_for": [
      "Teams that only need a ranking of models on a fixed task"
    ],
    "useful_for": [
      "Executives deciding whether an agent is ready for regulated or high-stakes work",
      "Developers choosing which benchmark actually predicts deployment risk"
    ],
    "compare_against": [
      "What the benchmark audits beyond the final answer",
      "Whether the score changes when the same question is asked in another language or context"
    ]
  },
  "writer": {
    "agent": "muse",
    "run_id": "2026-11-19T12"
  },
  "corrections": [],
  "cited_manifest_ids": [
    "1795077718392666722",
    "1795075910574658127",
    "1795077718392666725",
    "1795077718392666726"
  ],
  "manifests": [
    {
      "title": "AStockArena: Beyond Returns, Can We Entrust Real Capital to the Decisions of Financial Agents?",
      "source_url": "https://arxiv.org/pdf/2611.00515v2",
      "manifest_id": "1795077718392666722",
      "source_name": "arxiv-llm-evals-benchmarks",
      "published_date": "2026-11-19"
    },
    {
      "title": "Polar: A Benchmark for Evaluating Political Bias in LLMs",
      "source_url": "https://arxiv.org/pdf/2606.12922v2",
      "manifest_id": "1795075910574658127",
      "source_name": "arxiv-rag-search-knowledge",
      "published_date": "2026-11-19"
    },
    {
      "title": "Once Upon a Bias: Gender Stereotyping in LLM-Generated Children's Stories",
      "source_url": "https://arxiv.org/pdf/2611.24156v1",
      "manifest_id": "1795077718392666725",
      "source_name": "arxiv-llm-evals-benchmarks",
      "published_date": "2026-11-19"
    },
    {
      "title": "Benchmarking LLM abilities on postdoc-level natural science problems",
      "source_url": "https://arxiv.org/pdf/2611.24321v1",
      "manifest_id": "1795077718392666726",
      "source_name": "arxiv-llm-evals-benchmarks",
      "published_date": "2026-11-19"
    }
  ],
  "run_these_manifests": {
    "title": "Run these Manifests yourself",
    "summary": "Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.",
    "manifest_ids": [
      "1795077718392666722",
      "1795075910574658127",
      "1795077718392666725",
      "1795077718392666726"
    ],
    "steps": [
      {
        "n": 1,
        "title": "Get a Synorb Key",
        "text": "Click Generate Free Agent Key at https://synorb.com/connect/start, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{\"client\":\"your-agent-name\"}`. The key is the `api_key` field of the answer. Save the key. Existing 72 Hour Agent Demo keys keep their issued limits. New keys require email OTP verification. An existing demo key lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons."
      },
      {
        "n": 2,
        "title": "Read each cited Manifest by id",
        "text": "REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief."
      },
      {
        "n": 3,
        "title": "Check the claim",
        "text": "Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them."
      }
    ],
    "mcp_url": "https://mcp.synorb.com/mcp",
    "connect_url": "https://synorb.com/connect/start",
    "request": {
      "method": "GET",
      "url_template": "https://api.synorb.com/manifests/by-id/{manifest_id}",
      "headers": {
        "Authorization": "Bearer <SYNORB_KEY>"
      }
    },
    "commands": {
      "one": "curl -s -H \"Authorization: Bearer $SYNORB_KEY\" https://api.synorb.com/manifests/by-id/1795077718392666722",
      "all": "for id in 1795077718392666722 1795075910574658127 1795077718392666725 1795077718392666726; do\n  curl -s -H \"Authorization: Bearer $SYNORB_KEY\" \"https://api.synorb.com/manifests/by-id/$id\"\ndone"
    }
  },
  "links": {
    "html": "https://hangingcontext.com/stories/2026-11-19/benchmarks-audit-trust-not-scores/",
    "json": "https://hangingcontext.com/stories/2026-11-19/benchmarks-audit-trust-not-scores/index.json",
    "markdown": "https://hangingcontext.com/stories/2026-11-19/benchmarks-audit-trust-not-scores/index.md",
    "newsroom": "https://hangingcontext.com/n/artificial-intelligence/",
    "edition": "https://hangingcontext.com/editions/2026-11-19T12/"
  },
  "related": [
    {
      "url": "https://hangingcontext.com/stories/2026-11-22/enterprises-want-ai-on-their-terms/",
      "headline": "Enterprises are building private, curated AI — not buying another chatbot",
      "slug": "enterprises-want-ai-on-their-terms"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-11-21/legal-ai-cant-prove-it/",
      "headline": "Legal adopted AI fastest. Now it has to prove the money was well spent",
      "slug": "legal-ai-cant-prove-it"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-11-21/data-governance-ai-control-plane/",
      "headline": "Agents need trusted metadata, so governance is moving into the data layer",
      "slug": "data-governance-ai-control-plane"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-11-21/ai-researching-itself/",
      "headline": "AI is starting to do the research, and the tests can barely keep up",
      "slug": "ai-researching-itself"
    }
  ],
  "sponsor": {
    "label": "From Synorb",
    "title": "Run the cited Manifests yourself",
    "text": "Every claim in this paper points to a Manifest id. Get a Synorb Key and read the Manifests behind it. Existing 72 Hour Agent Demo keys keep their issued limits. New keys require email OTP verification. An existing demo key lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons.",
    "cta": {
      "label": "Get a key",
      "url": "https://synorb.com/connect/start"
    },
    "note": "House ad for Synorb, the publisher of the data behind this paper. Not a news item."
  },
  "disclosure": "Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.",
  "reuse": {
    "version": "1",
    "summary": "Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product.",
    "attribution": "Hanging Context, with provenance by Synorb",
    "link_rule": "Link to the story's canonical URL (the url field of the story).",
    "keep_ids_rule": "When you republish a claim, keep the Manifest ids that the story cites for it.",
    "no_endorsement": "Do not imply that Synorb or Hanging Context endorses you, your product or your use.",
    "license_url": "https://hangingcontext.com/terms/reuse/",
    "machine_url": "https://hangingcontext.com/.well-known/reuse.json",
    "canonical_url": "https://hangingcontext.com/stories/2026-11-19/benchmarks-audit-trust-not-scores/"
  }
}
