AI can read the filings now. The catch is checking its arithmetic.
Small open models extract figures from SEC filings with accuracy ranging from 17% to 83%. A financial research agent gets 29% of calculations right on its own — and 100% when it uses Python.
- In a preprint, small open models of 7 to 21 billion parameters scored between 17.1% and 82.7% extracting figures from SEC filings in plain setups, and automated verification failed to catch 16 to 167 wrong answers per 1,000. Self-reported m:…0696
- In a preprint, Mistral-7B achieved only 29% accuracy on financial calculations, compared with 100% for Python-based computation. Self-reported m:…0712
- In a preprint, independent evidence verification caught 77% of errors, against 35% for checks that shared the writer's evidence Self-reported m:…0712
- In a preprint, explanatory layoff text increased Mistral's market participation by 5.3 times, while evaluative text lowered average participation and executed net flow across all tested model families. Self-reported m:…8863
The extraction test
The first question for any team that wants AI to read financial documents is a simple one: can a small, cheap, open model pull the right number out of a filing? A preprint posted this week introduces Filing-Evidence, a benchmark for evaluating small open-source language models on extracting verifiable data from SEC filings. The answer is that it depends enormously on the model. Small open models of 7 to 21 billion parameters showed accuracy ranging from 17.1% to 82.7% in plain setups — a spread wide enough that the choice of model matters more than the choice of pipeline. m:…0696
The more important finding is about the checking, not the reading. Automated verification protocols failed to catch 16 to 167 wrong answers per 1,000, and 57% of those missed errors came from scale or unit mismatches — a model reporting millions as billions, or the wrong year's figure. Reasoning-based inference modes significantly reduced scale and temporal errors for four out of five tested models. In other words, the model can be taught to be more careful, but the verification layer still lets a meaningful share of wrong answers through, and the errors it misses are exactly the kind that look plausible in a spreadsheet. m:…0696
The arithmetic test
A second preprint, FinVeriQA, tests a financial research agent end to end — routing a question, retrieving evidence, doing the maths, and verifying the answer. Its bluntest result is about arithmetic: Mistral-7B achieved only 29% accuracy on financial calculations, compared with 100% for Python-based computation. The paper's conclusion is that a language model should not be doing the sums at all; the agent should route calculations to code, and the model's job is to decide what to calculate. m:…0712
The same paper contains a warning about verification that every buyer of a financial agent should read twice. Independent evidence verification — checking the answer against the original passage — caught 77% of errors, compared with 35% for checks that shared the evidence the writer had seen. A check that reuses the writer's evidence is mostly checking the writer's homework against itself. m:…0712
There is a constructive result too. The paper's modular architecture separates routing, computation and evidence verification, and its compact routing model, Laya, beat Mistral-7B on routing accuracy, 0.86 against 0.68. The lesson is architectural rather than heroic: small, specialised components, each doing one job, beat a single model asked to do everything. m:…0712
The framing test
A third preprint this week asks what happens when these agents start trading rather than just reading. Researchers tested multiple model families as trading agents and found that the framing of corporate announcements significantly impacts their decisions — even when the underlying facts are identical. Explanatory layoff text increased Mistral's market participation by 5.3 times, while evaluative text lowered average participation and executed net flow across all tested model families. m:…8863
The subtlest finding is that active composition can change significantly while net flow remains stable, because different model families respond to the same framing in offsetting ways. A market full of these agents could look calm in aggregate while churning underneath — the total flow barely moves, but who is buying and selling has changed completely. For anyone running a trading desk, that is the number to watch: not whether the agents agree with each other, but whether they are all reacting to the same words in the same way at the same time. m:…8863
What is still unresolved
- Whether the Filing-Evidence accuracy spread narrows with better prompting or retrieval, or whether it is a property of the models themselves.
- Whether FinVeriQA's modular architecture holds up on live financial data rather than benchmark questions.
- How widely trading agents are actually deployed, and whether framing effects show up in real market data.
- What verification standard regulators or auditors will require before model-extracted figures can be used in filings and reports.
Provenance by Synorb. Every claim links to its source. This story was written by an agent from the Manifests listed below. Read the methodology and limitations, or send a correction.
Free to reuse, including commercially. Credit “Hanging Context, with provenance by Synorb” and link to this story. Keep the cited Manifest ids. Reuse terms v1.