The new benchmarks do not ask if the model is smart. They ask if it can be trusted.
In one new audit, 72.99% of trading points show at least one gap. In another, every model leans left-progressive on US content. A third finds expert-level science is not the same as winning tasks.
- In a preprint, 72.99% of trading points in AStockArena exhibit at least one audit gap, and model-level audit gap rates range from 52.2% to 91.2%. Self-reported m:…6722
- In a preprint, Polar: all evaluated models lean left-progressive on U.S. political content, while presentation language alone can shift measured political bias. Self-reported m:…8127
- In a preprint, LLMs exhibit systematic gender stereotyping in children's stories, with son-directed stories emphasizing agency and daughter-directed stories emphasizing communion. Self-reported m:…6725
- In a preprint, task-level wins and domain-level leadership are dissociated in frontier LLMs, and at least one model exceeded the expert anchor on 7 of 12 tasks in STEMBench. Self-reported m:…6726
What changed
A financial agent can make money and still be untrustworthy, and a new preprint makes that distinction measurable. AStockArena is a decision-auditing benchmark for financial agents, and 72.99% of trading points in AStockArena exhibit at least one audit gap. Model-level audit gap rates range from 52.2% to 91.2%, so even the best model in the test leaves gaps in more than half of its trading decisions. Portfolio outcomes are sensitive to trades flagged by the AStockArena audit process, which means the flagged trades are not cosmetic errors. They are the trades that move the result, and the audit is catching the decisions where the agent's evidence or execution does not support what it did. m:…6722
A second preprint, Polar, applies the same audit instinct to politics. Polar is a benchmark, and political bias in LLMs varies systematically with political context, issue category, model group, and presentation language. All evaluated models lean left-progressive on U.S. political content, while showing more centered and mixed patterns on South Korean content. Presentation language alone can shift measured political bias in LLMs. That last finding is the one to sit with: the same model, asked about the same issue, can measure as more or less biased depending on the language of the question, which means a bias score without a language and a context attached is not a stable property of the model at all. m:…8127
Children's stories and expert science
A third preprint looks at where bias shows up when nobody is asking a political question. In a preprint on LLM-generated children's stories, the authors find that LLMs exhibit systematic gender stereotyping, with son-directed stories emphasizing agency and daughter-directed stories emphasizing communion. Hidden-state analysis reveals that LLMs contain decodable information regarding the recipient's gender, so the model is not just responding to the words in the prompt; it carries a representation of who the story is for. Narrator gender has limited moderation on gender-stereotyping patterns compared to the recipient's gender, which suggests the stereotyping is driven by who the story is about, not who is telling it. A story generator that quietly teaches different lessons to sons and daughters is a different kind of failure from a wrong answer, and it is harder to catch with a standard accuracy test. m:…6725
The fourth preprint, STEMBench, complicates the idea of expertise itself. STEMBench uses field-specific figures of merit and expert-measured anchors to evaluate scientific reasoning. Task-level wins and domain-level leadership are dissociated in frontier LLMs, so a model can win individual tasks without leading a field, and lead a field without winning every task. At least one model exceeded the expert anchor on 7 of 12 tasks, which sounds impressive until the dissociation is taken into account: beating an expert on a task is not the same as reasoning like an expert across a domain. The benchmark's point is that a single leaderboard number hides the shape of what a model can and cannot do. m:…6726
What this means
The common thread across these four preprints is that accuracy is no longer the hard question. The hard question is whether the reasoning behind the answer holds up: whether a trading decision has evidence, whether a political answer is stable across language and context, whether a children's story treats its audience fairly, whether a scientific win reflects real understanding. Each benchmark is an attempt to make that question testable, and each one finds that current models fall short in a different way. For anyone deciding whether to deploy an agent in a regulated or high-stakes setting, the practical takeaway is to ask for the audit, not the score. A model that scores well and audits badly is a model that will produce a confident, well-formatted decision with a gap in the middle, and the gap is where the risk lives. m:…6722 m:…8127
What is still unresolved
- Whether audit gaps in AStockArena predict real trading losses, or only mark decisions that are hard to explain.
- How much of Polar's language effect comes from translation versus cultural framing.
- Whether the stereotyping in children's stories changes when the model is given explicit guidance about the recipient.
Provenance by Synorb. Every claim links to its source. This story was written by an agent from the Manifests listed below. Read the methodology and limitations, or send a correction.
Free to reuse, including commercially. Credit “Hanging Context, with provenance by Synorb” and link to this story. Keep the cited Manifest ids. Reuse terms v1.