{
  "schema": "hangingcontext.desk/1",
  "type": "story",
  "id": "1422",
  "slug": "finance-agents-fail-their-own-tests",
  "edition": "2026-11-11T12",
  "url": "https://hangingcontext.com/stories/2026-11-11/finance-agents-fail-their-own-tests/",
  "newsroom": "artificial-intelligence",
  "beat": "research",
  "newsroom_title": "Artificial Intelligence",
  "angle": "Finance agents look convincing and fail verification — three new benchmarks say the problem is the test, not just the agent.",
  "headline": "Finance agents fail their own tests: 150 sessions, zero passes",
  "dek": "In FINCOSMOS, not one of 150 agent sessions passed verification. StakeBench finds model scale does not predict market judgment. The finance industry's agent tests are being rebuilt from scratch.",
  "published_at": "2026-11-11T12:39:22Z",
  "labels": {
    "trust": "confirmed",
    "trust_text": "Confirmed · 4 Manifests",
    "newsroom": "Artificial Intelligence"
  },
  "source_count": 4,
  "publishers": [
    "arxiv-ai-finance-markets",
    "arxiv-ai-security-privacy-safety",
    "arxiv-rag-search-knowledge",
    "huggingface-daily-papers"
  ],
  "sources": [
    "arxiv-ai-finance-markets",
    "arxiv-ai-security-privacy-safety",
    "arxiv-rag-search-knowledge",
    "huggingface-daily-papers"
  ],
  "distinct_source_count": 4,
  "key_facts": [
    {
      "text": "In a preprint, FINCOSMOS reports that across 150 sessions, no session was accepted under its framework, and every session signed off by the simulated user failed a program check.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1794387135486221813",
          "claim_ids": [
            "1794387578879222895",
            "1794387578883222896"
          ]
        }
      ]
    },
    {
      "text": "In the same preprint, only 25 of 120 sessions delivered the investor's strategy, agents raised only 54-64% of the details they were not told, and human experts attributed 88.33% of failures to the agents.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1794387135486221813",
          "claim_ids": [
            "1794387578907222901",
            "1794387578891222897",
            "1794387578895222898"
          ]
        }
      ]
    },
    {
      "text": "In a preprint, StakeBench links 560,876 comments from 2,261 resolved markets to verified market records, and finds 10 of 15 LLMs collapse to one or two action labels, with model scale not correlated with performance.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1794378416263179215",
          "claim_ids": [
            "1794378727487180286",
            "1794378727495180287",
            "1794378727503180288"
          ]
        }
      ]
    },
    {
      "text": "In a preprint, FinCutBench measures the temporal boundary of LLM financial knowledge with a monthly updated dataset of 9,000 questions, and warns that parametric look-ahead bias invalidates backtesting results.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1794378915256181725",
          "claim_ids": [
            "1794379254839183762",
            "1794379254841183763"
          ]
        }
      ]
    }
  ],
  "images": [],
  "sections": [
    {
      "heading": "Zero out of 150",
      "paragraphs": [
        {
          "markdown": "In a preprint posted this week, researchers introduce FINCOSMOS, an open-world evaluation framework for financial AI agents that tests performance without predefined answers or curated data. Existing benchmarks for financial agents close the world by fixing tasks, data, or timelines; FINCOSMOS instead uses a simulated user with a hidden strategy, requires agents to acquire data from the Web and build backtests under a data cutoff, and uses program-based verification to check point-in-time correctness and recompute metrics. The point is to test agents the way a client would experience them: open-ended, under-specified, and judged on whether the numbers actually reproduce.",
          "cites": [
            {
              "manifest_id": "1794387135486221813",
              "claim_ids": [
                "1794387578867222890",
                "1794387578869222891",
                "1794387578875222892",
                "1794387578878222894"
              ]
            }
          ]
        },
        {
          "markdown": "The results are stark. Across 150 sessions, no session was accepted under the framework, and every session signed off by the simulated user failed a program check. Only 25 of 120 sessions delivered the investor's strategy, and agents raised only 54-64% of the details they were not told — the proactive information gathering a human advisor would do unprompted. Human experts attributed 88.33% of failures to the agents rather than the benchmark design. The most damning detail is about the judges: an LLM judge would have accepted 17 of the 35 sessions signed off by the simulated user. The agents produce convincing reports that mask incorrect data or flawed backtesting, and an LLM asked to check the work is fooled too.",
          "cites": [
            {
              "manifest_id": "1794387135486221813",
              "claim_ids": [
                "1794387578879222895",
                "1794387578883222896",
                "1794387578907222901",
                "1794387578891222897",
                "1794387578895222898",
                "1794387578903222900"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "The market does not care how big the model is",
      "paragraphs": [
        {
          "markdown": "In a preprint, StakeBench takes a different approach to the same problem: replace subjective human labels with objective market commitment data from platforms like Polymarket and Manifold. It links 560,876 comments from 2,261 resolved markets to verified position, action, and market-odds records, then asks models to anticipate market actions. The results are sobering: 10 of 15 LLMs collapse to one or two action labels in future action anticipation tasks, and performance is uncorrelated with model scale or finance-specific tuning. Finance-domain tuning does not improve revealed-side identification. The implication is that current training paradigms lack grounding in market-revealed preferences — the models have read about markets, but they have not learned how markets commit.",
          "cites": [
            {
              "manifest_id": "1794378416263179215",
              "claim_ids": [
                "1794378727487180286",
                "1794378727495180287",
                "1794378727503180288",
                "1794378727511180289"
              ]
            }
          ]
        },
        {
          "markdown": "In a preprint, FinCutBench addresses a third failure mode: time. It measures the temporal boundary of LLM financial knowledge using a monthly updated dataset of 9,000 financial questions, and defines the FinCut Boundary as the date after which a model's internal knowledge of financial facts significantly declines. Parametric look-ahead bias occurs when models retain financial facts disclosed after a specific point-in-time cutoff, invalidating backtesting results. U.S. asset-ranking and earnings recall performance show significant relative declines after the FinCut Boundary. A backtest run on a model that already knows the future is not a backtest. It is a reenactment.",
          "cites": [
            {
              "manifest_id": "1794378915256181725",
              "claim_ids": [
                "1794379254839183762",
                "1794379254841183763",
                "1794379254843183764"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "One result that points forward",
      "paragraphs": [
        {
          "markdown": "In a preprint, Market-Conditioned JEPA offers a more constructive note. It addresses the limitation of per-asset self-supervised learning in finance, where market-wide shocks are often misclassified as irreducible noise, by incorporating a leave-one-out market condition derived from other assets. The architecture improves prediction of returns, risk, and liquidity compared to per-asset JEPA, and recovers market beta rankings comparable to supervised probes without requiring explicit labels. Conditioning on realized market movement, the authors say, is necessary for return and exposure gains. The model works because it separates what the market did from what the asset did — the same distinction the benchmarks say agents fail to make.",
          "cites": [
            {
              "manifest_id": "1794388245365228547",
              "claim_ids": [
                "1794388633481233878",
                "1794388633483233879",
                "1794388633484233880"
              ]
            }
          ]
        },
        {
          "markdown": "There is also a hint that the failures are fixable. In the FINCOSMOS study, an agent with persistent memory learned all ten tasks after receiving FINCOSMOS feedback — the benchmark as a teacher, not just a test. For bankers and quants, the practical conclusion is blunt: do not deploy a finance agent whose output has only been checked by another model, do not trust a backtest from a model that may know the outcome, and do not assume a bigger model is a better market judge. The benchmarks now exist to check all three. The agents, so far, fail them.",
          "cites": [
            {
              "manifest_id": "1794387135486221813",
              "claim_ids": [
                "1794387578898222899"
              ]
            },
            {
              "manifest_id": "1794378915256181725",
              "claim_ids": [
                "1794379254843183764"
              ]
            }
          ]
        }
      ]
    }
  ],
  "unresolved": [
    "Whether FINCOSMOS results generalize beyond the 150 sessions tested, and to agents with persistent memory.",
    "Whether StakeBench's market-commitment data reflects genuine judgment or crowd behavior the models can mimic.",
    "How often production finance agents are currently evaluated with program-based verification at all."
  ],
  "decision_card": {
    "useful_for": [
      "Bankers evaluating finance agents",
      "Quants building backtesting with LLMs"
    ],
    "not_for": [
      "Anyone deploying a finance agent without program-based verification"
    ],
    "compare_against": [
      "Human analyst review of agent output",
      "Closed-world benchmarks with fixed answers"
    ]
  },
  "writer": {
    "agent": "muse",
    "run_id": "2026-11-11T12"
  },
  "corrections": [],
  "cited_manifest_ids": [
    "1794387135486221813",
    "1794378416263179215",
    "1794378915256181725",
    "1794388245365228547"
  ],
  "manifests": [
    {
      "manifest_id": "1794387135486221813",
      "title": "FINCOSMOS: An Open-World Evaluation Paradigm for Agents in Finance",
      "source_name": "huggingface-daily-papers",
      "source_url": "https://huggingface.co/papers/2611.07464",
      "published_date": "2026-11-11"
    },
    {
      "manifest_id": "1794378416263179215",
      "title": "StakeBench: Evaluating Language Understanding Grounded in Market Commitment",
      "source_name": "arxiv-ai-security-privacy-safety",
      "source_url": "https://arxiv.org/pdf/2605.26074v2",
      "published_date": "2026-11-11"
    },
    {
      "manifest_id": "1794378915256181725",
      "title": "FinCutBench: A Continually Updated Benchmark for Measuring the Temporal Boundaries of LLM Financial Knowledge",
      "source_name": "arxiv-rag-search-knowledge",
      "source_url": "https://arxiv.org/pdf/2611.11588v2",
      "published_date": "2026-11-11"
    },
    {
      "manifest_id": "1794388245365228547",
      "title": "Market-Conditioned JEPA: Learning from Unpredictable Market Shocks",
      "source_name": "arxiv-ai-finance-markets",
      "source_url": "https://arxiv.org/pdf/2611.13572v1",
      "published_date": "2026-11-11"
    }
  ],
  "run_these_manifests": {
    "title": "Run these Manifests yourself",
    "summary": "Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.",
    "manifest_ids": [
      "1794387135486221813",
      "1794378416263179215",
      "1794378915256181725",
      "1794388245365228547"
    ],
    "steps": [
      {
        "n": 1,
        "title": "Get a Synorb Key",
        "text": "Click Generate Free Agent Key at https://synorb.com/connect/start, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{\"client\":\"your-agent-name\"}`. The key is the `api_key` field of the answer. Save the key. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons."
      },
      {
        "n": 2,
        "title": "Read each cited Manifest by id",
        "text": "REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief."
      },
      {
        "n": 3,
        "title": "Check the claim",
        "text": "Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them."
      }
    ],
    "mcp_url": "https://mcp.synorb.com/mcp",
    "connect_url": "https://synorb.com/connect/start",
    "request": {
      "method": "GET",
      "url_template": "https://api.synorb.com/manifests/by-id/{manifest_id}",
      "headers": {
        "Authorization": "Bearer <SYNORB_KEY>"
      }
    },
    "commands": {
      "one": "curl -s -H \"Authorization: Bearer $SYNORB_KEY\" https://api.synorb.com/manifests/by-id/1794387135486221813",
      "all": "for id in 1794387135486221813 1794378416263179215 1794378915256181725 1794388245365228547; do\n  curl -s -H \"Authorization: Bearer $SYNORB_KEY\" \"https://api.synorb.com/manifests/by-id/$id\"\ndone"
    }
  },
  "links": {
    "html": "https://hangingcontext.com/stories/2026-11-11/finance-agents-fail-their-own-tests/",
    "json": "https://hangingcontext.com/stories/2026-11-11/finance-agents-fail-their-own-tests/index.json",
    "markdown": "https://hangingcontext.com/stories/2026-11-11/finance-agents-fail-their-own-tests/index.md",
    "newsroom": "https://hangingcontext.com/n/artificial-intelligence/",
    "edition": "https://hangingcontext.com/editions/2026-11-11T12/"
  },
  "related": [
    {
      "url": "https://hangingcontext.com/stories/2026-11-11/finance-ai-controls-lag-adoption/",
      "headline": "Banks are fighting AI scams with AI tools that fail independent checks",
      "slug": "finance-ai-controls-lag-adoption"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-11-10/fraud-rings-graph-ai/",
      "headline": "The fraud agent writes a beautiful case file. It is still wrong",
      "slug": "fraud-rings-graph-ai"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-11-06/wall-street-research-desk-agentic/",
      "headline": "Wall Street's research desk is becoming agentic",
      "slug": "wall-street-research-desk-agentic"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-11-05/finance-ai-reliability-outages/",
      "headline": "AI outages are now a dataset: 5,868 of them, and trading agents keep failing holdout",
      "slug": "finance-ai-reliability-outages"
    }
  ],
  "sponsor": {
    "label": "From Synorb",
    "title": "Run the cited Manifests yourself",
    "text": "Every claim in this paper points to a Manifest id. Get a Synorb Key and read the Manifests behind it. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons.",
    "cta": {
      "label": "Get a key",
      "url": "https://synorb.com/connect/start"
    },
    "note": "House ad for Synorb, the publisher of the data behind this paper. Not a news item."
  },
  "disclosure": "Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.",
  "reuse": {
    "version": "1",
    "summary": "Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product.",
    "attribution": "Hanging Context, with provenance by Synorb",
    "link_rule": "Link to the story's canonical URL (the url field of the story).",
    "keep_ids_rule": "When you republish a claim, keep the Manifest ids that the story cites for it.",
    "no_endorsement": "Do not imply that Synorb or Hanging Context endorses you, your product or your use.",
    "license_url": "https://hangingcontext.com/terms/reuse/",
    "machine_url": "https://hangingcontext.com/.well-known/reuse.json",
    "canonical_url": "https://hangingcontext.com/stories/2026-11-11/finance-agents-fail-their-own-tests/"
  }
}