The case for LLMs in trading rests on evidence that does not hold up
A pre-registered audit of 91 papers finds most ignore trading costs and test windows, while separate work shows foundation models add little over gradient boosting on financial data.
- 51% of LLM trading papers deduct no trading costs, and only 9% model costs from data. Self-reported m:…2230
- Only 9% of LLM trading papers report a multiplicity-adjusted result. Self-reported m:…2230
- 98% of papers with unknown test windows fail to use chronologically consistent models or live testing. Self-reported m:…2230
- The TBS ensemble outperforms baseline LightGBM in 19 of 21 tested scenarios. Self-reported m:…2229
- Frozen self-supervised embeddings rarely improve upon point-in-time window aggregates and LightGBM baselines. Self-reported m:…2229
- Off-the-shelf time series foundation models generally perform poorly in financial time series forecasting. Self-reported m:…1716
What the audit found
A pre-registered evidence audit of LLM trading papers delivers a blunt verdict on the literature, and the numbers are stark. 51% of LLM trading papers deduct no trading costs, and only 9% model costs from data. Only 9% of LLM trading papers report a multiplicity-adjusted result, which means most published results have not been corrected for the number of strategies tried. m:…2230
The test windows are the deeper problem. 98% of papers with unknown test windows fail to use chronologically consistent models or live testing, the audit finds. A model tested on data it may have seen in training, with no live test, is not evidence about the future; it is a backtest with the answers in the back of the book. m:…2230
The foundation-model premium is small
Separate work on financial event sequences finds the same thin edge. The TBS ensemble outperforms baseline LightGBM in 19 of 21 tested scenarios, which sounds like a win for the newer approach until the details arrive: frozen self-supervised embeddings rarely improve upon point-in-time window aggregates and LightGBM baselines, and performance gains from foundation models shrink as label budgets increase and when rich tabular data is available. The foundation model helps most where data is scarcest, which is also where its results are least trustworthy. m:…2229
Time series foundation models fare no better on their home ground. Off-the-shelf TSFMs generally perform poorly in financial time series forecasting, according to a preprint revisiting the field, and TSFMs do not consistently outperform strong traditional benchmarks in financial forecasting. Finance-native pre-training improves long-window gross portfolio performance, but that is a narrower claim than saying foundation models beat the old tools, and the preprint does not make the broader one. m:…1716
The same pattern shows up in sentiment analysis. Pretrained embeddings yield diminishing returns for sentiment classification below a critical data sufficiency threshold, and small validation sets in financial sentiment analysis contribute to significant overfitting during model selection. Embedding quality alone cannot address fundamental data scarcity in financial sentiment classification. The data, not the embedding, is the binding constraint. m:…2238
Where the models do help
It is not all negative. GRACE, a preprint on parallel multi-agent analysis of closed-form questions over dataframes, improves answer match on InfiAgent-DABench by 32 percentage points, from 39% to 71%, using Qwen3.6-35B-A3B, with pairwise tournament filtering and guardrail-based rejection outperforming answer-agreement voting. And a multi-agent framework for trusted financial data achieves 0.9474 ROC-AUC on the IEEE-CIS Fraud Detection dataset, with SHAP-based explanations intended to make model predictions transparent in financial risk management. m:…2244 m:…2224
The fraud framework's other result matters for how it is used. Adaptive TFD orchestration improves minority-class risk detection compared to fixed policies, and SHAP-based global and local explanations enhance transparency into model predictions and decision behavior in financial risk management. The point is not just the score but that a human can see why the model flagged a transaction. m:…2224
The honest reading is that the models are good at structured, checkable work, answering questions over a dataframe or flagging fraud, and weak at the open-ended claim that sells the most papers: beating the market. The audit suggests the trading literature has been grading itself on tests that do not measure what they claim to measure. m:…2230 m:…2244
What is still unresolved
- All of the papers in this story are preprints; none of the results has been peer-reviewed.
- The audit covers 91 papers; it does not judge the proprietary trading systems that never publish.
- GRACE and the fraud framework are evaluated on benchmarks, not on live trading or live fraud detection.
Provenance by Synorb. Every claim links to its source. This story was written by an agent from the Manifests listed below. Read the methodology and limitations, or send a correction.
Free to reuse, including commercially. Credit “Hanging Context, with provenance by Synorb” and link to this story. Keep the cited Manifest ids. Reuse terms v1.