Finance agents fail their own tests: 150 sessions, zero passes

Zero out of 150

In a preprint posted this week, researchers introduce FINCOSMOS, an open-world evaluation framework for financial AI agents that tests performance without predefined answers or curated data. Existing benchmarks for financial agents close the world by fixing tasks, data, or timelines; FINCOSMOS instead uses a simulated user with a hidden strategy, requires agents to acquire data from the Web and build backtests under a data cutoff, and uses program-based verification to check point-in-time correctness and recompute metrics. The point is to test agents the way a client would experience them: open-ended, under-specified, and judged on whether the numbers actually reproduce. m:…1813

The results are stark. Across 150 sessions, no session was accepted under the framework, and every session signed off by the simulated user failed a program check. Only 25 of 120 sessions delivered the investor's strategy, and agents raised only 54-64% of the details they were not told — the proactive information gathering a human advisor would do unprompted. Human experts attributed 88.33% of failures to the agents rather than the benchmark design. The most damning detail is about the judges: an LLM judge would have accepted 17 of the 35 sessions signed off by the simulated user. The agents produce convincing reports that mask incorrect data or flawed backtesting, and an LLM asked to check the work is fooled too. m:…1813

The market does not care how big the model is

In a preprint, StakeBench takes a different approach to the same problem: replace subjective human labels with objective market commitment data from platforms like Polymarket and Manifold. It links 560,876 comments from 2,261 resolved markets to verified position, action, and market-odds records, then asks models to anticipate market actions. The results are sobering: 10 of 15 LLMs collapse to one or two action labels in future action anticipation tasks, and performance is uncorrelated with model scale or finance-specific tuning. Finance-domain tuning does not improve revealed-side identification. The implication is that current training paradigms lack grounding in market-revealed preferences — the models have read about markets, but they have not learned how markets commit. m:…9215

In a preprint, FinCutBench addresses a third failure mode: time. It measures the temporal boundary of LLM financial knowledge using a monthly updated dataset of 9,000 financial questions, and defines the FinCut Boundary as the date after which a model's internal knowledge of financial facts significantly declines. Parametric look-ahead bias occurs when models retain financial facts disclosed after a specific point-in-time cutoff, invalidating backtesting results. U.S. asset-ranking and earnings recall performance show significant relative declines after the FinCut Boundary. A backtest run on a model that already knows the future is not a backtest. It is a reenactment. m:…1725

One result that points forward

In a preprint, Market-Conditioned JEPA offers a more constructive note. It addresses the limitation of per-asset self-supervised learning in finance, where market-wide shocks are often misclassified as irreducible noise, by incorporating a leave-one-out market condition derived from other assets. The architecture improves prediction of returns, risk, and liquidity compared to per-asset JEPA, and recovers market beta rankings comparable to supervised probes without requiring explicit labels. Conditioning on realized market movement, the authors say, is necessary for return and exposure gains. The model works because it separates what the market did from what the asset did — the same distinction the benchmarks say agents fail to make. m:…8547

There is also a hint that the failures are fixable. In the FINCOSMOS study, an agent with persistent memory learned all ten tasks after receiving FINCOSMOS feedback — the benchmark as a teacher, not just a test. For bankers and quants, the practical conclusion is blunt: do not deploy a finance agent whose output has only been checked by another model, do not trust a backtest from a model that may know the outcome, and do not assume a bigger model is a better market judge. The benchmarks now exist to check all three. The agents, so far, fail them. m:…1813 m:…1725