{
  "schema": "hangingcontext.desk/1",
  "type": "story",
  "id": "76",
  "slug": "llms-meet-finance",
  "edition": "2026-10-15T12",
  "url": "https://hangingcontext.com/stories/2026-10-15/llms-meet-finance/",
  "newsroom": "artificial-intelligence",
  "beat": "research",
  "newsroom_title": "Artificial Intelligence",
  "audience": "banker",
  "angle": "Finance stress-tests LLMs: hallucinations in financial QA and quant evals",
  "headline": "Finance is stress-testing LLMs and finding confident hallucinations",
  "dek": "Four new preprints put LLMs through finance's gauntlet: 15-23% of confident answers are wrong on FinQA, a 1,575-instance benchmark grades models by backtest rather than syntax, and proprietary models still lead open-weight ones 92.8% to 84.1% on accounting-relation extraction.",
  "published_at": "2026-10-15T12:45:49Z",
  "labels": {
    "trust": "confirmed",
    "trust_text": "Confirmed · 4 sources",
    "newsroom": "Artificial Intelligence"
  },
  "source_count": 4,
  "publishers": [
    "arxiv-ai-finance-markets",
    "arxiv-llm-evals-benchmarks",
    "arxiv-rag-search-knowledge"
  ],
  "key_facts": [
    {
      "text": "In a preprint, the authors report that 15-23% of confident answers (where 8 resamples agree) are wrong on the FinQA benchmark, and that linear probes on the residual stream significantly outperform output-level baselines (log-probabilities and self-assessment) in detecting confident hallucinations.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1792056054187313827",
          "claim_ids": [
            "1792056347387119330",
            "1792056347378098277"
          ]
        }
      ]
    },
    {
      "text": "In a preprint, QuantEval's authors describe a 1,575-instance expert-validated benchmark across knowledge-based QA, quantitative mathematical reasoning, and quantitative strategy coding, graded with deterministic CTA-style backtesting rather than syntax or template matching; they report domain-aligned SFT and RL improve quantitative reasoning but trade off factual recall.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1792051595356674784",
          "claim_ids": [
            "1792052043959138919",
            "1792052043962076643",
            "1792052043966650666"
          ]
        }
      ]
    },
    {
      "text": "In a preprint, the authors report proprietary models outperform open-weight models in financial relation classification, achieving 92.8% accuracy compared to 84.1%, on schema-guided evidence discovery over SEC 10-K filings.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1792044246539544366",
          "claim_ids": [
            "1792044583447388713",
            "1792044583449253517",
            "1792044583450529985"
          ]
        }
      ]
    },
    {
      "text": "In a preprint, AlphaRLM's authors report it outperforms strong factor-mining and LLM-based baselines in annualized excess return across CSI 300, CSI 1000, and S&P 500.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1792044246539159128",
          "claim_ids": [
            "1792044580900900289"
          ]
        }
      ]
    }
  ],
  "images": [],
  "sections": [
    {
      "heading": "Confidently wrong",
      "paragraphs": [
        {
          "markdown": "The most alarming number in this batch: 15-23% of confident answers — defined as cases where 8 resamples agree — are wrong on the FinQA benchmark. These are not hedged guesses; they are the answers the model is surest about. The paper's proposed defense: linear probes on the residual stream significantly outperform output-level baselines (log-probabilities and self-assessment) in detecting confident hallucinations. Read the model's insides, not its confidence.",
          "cites": [
            {
              "manifest_id": "1792056054187313827",
              "claim_ids": [
                "1792056347387119330",
                "1792056347378098277"
              ]
            }
          ]
        },
        {
          "markdown": "The 15-23% range is worth sitting with — our read. It means agreement across resamples — the standard cheap uncertainty filter — is screening out nothing. Consensus is not correctness, and in finance the confident wrong answer is the expensive one.",
          "cites": [
            {
              "manifest_id": "1792056054187313827",
              "claim_ids": [
                "1792056347387119330"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "Grade the strategy, not the syntax",
      "paragraphs": [
        {
          "markdown": "A second preprint, QuantEval, argues that the way we grade quant models is broken: syntax or template matching cannot tell you whether a generated strategy works. Its replacement is a 1,575-instance expert-validated benchmark across knowledge-based QA, quantitative mathematical reasoning, and quantitative strategy coding — and it grades the coding tasks with deterministic CTA-style backtesting, which the authors call a more effective evaluation method for financial strategy coding than syntax or template matching. The training finding: domain-aligned SFT and RL improve quantitative reasoning in LLMs but result in trade-offs regarding factual recall.",
          "cites": [
            {
              "manifest_id": "1792051595356674784",
              "claim_ids": [
                "1792052043959138919",
                "1792052043962076643",
                "1792052043966650666"
              ]
            }
          ]
        },
        {
          "markdown": "The factual-recall trade-off is the paper's sting — our read. The same domain training that makes the model better at quantitative reasoning makes it worse at remembering facts. Specialization has a price, and in finance the price is paid in the exact currency the first paper is warning about: confident wrongness.",
          "cites": [
            {
              "manifest_id": "1792051595356674784",
              "claim_ids": [
                "1792052043966650666"
              ]
            },
            {
              "manifest_id": "1792056054187313827",
              "claim_ids": [
                "1792056347387119330"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "The accounting layer",
      "paragraphs": [
        {
          "markdown": "A third preprint moves from generic QA to the structure of the filings themselves. Financial disclosures contain underlying double-entry accounting relationships that can be extracted using schema-guided evidence discovery, the authors write — and their benchmark tests exactly that. The headline gap: proprietary models outperform open-weight models in financial relation classification, achieving 92.8% accuracy compared to 84.1%. The authors also report substantial variation in extraction volume and edge distributions across different LLM families when applied to SEC 10-K filings — the models do not even agree on how much structure they see.",
          "cites": [
            {
              "manifest_id": "1792044246539544366",
              "claim_ids": [
                "1792044583449253517",
                "1792044583447388713",
                "1792044583450529985"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "Factors from the literature",
      "paragraphs": [
        {
          "markdown": "The fourth preprint, AlphaRLM, automates symbolic factor discovery by aligning LLM generation with financial literature: Literature-Guided Operator Assembly (LEGO) provides effective semantic priors for symbolic factor construction, and reinforcement learning with executable-validity and grouped-return-separation rewards aligns LLM generation policies with market feedback. The reported result: AlphaRLM outperforms strong factor-mining and LLM-based baselines in annualized excess return across CSI 300, CSI 1000, and S&P 500. This is research about evaluation and generation methods — not a strategy to trade.",
          "cites": [
            {
              "manifest_id": "1792044246539159128",
              "claim_ids": [
                "1792044580901993330",
                "1792044580902374284",
                "1792044580900900289"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "Our read",
      "paragraphs": [
        {
          "markdown": "Our read: finance is building its own LLM eval stack, and it looks nothing like the generic leaderboards — probes for hallucinations, deterministic backtests for generated strategies, schema-guided extraction for filings. The pattern: in a domain where a wrong number has a dollar sign attached, the field is converging on evaluation that mirrors the actual work.",
          "cites": [
            {
              "manifest_id": "1792056054187313827",
              "claim_ids": [
                "1792056347378098277"
              ]
            },
            {
              "manifest_id": "1792051595356674784",
              "claim_ids": [
                "1792052043962076643"
              ]
            },
            {
              "manifest_id": "1792044246539544366",
              "claim_ids": [
                "1792044583449253517"
              ]
            }
          ]
        },
        {
          "markdown": "The 92.8-vs-84.1 gap is also a reminder — our read — that in finance, unlike in chat, the open-weight frontier still trails, and the gap is measured on the boring, structured work that actually matters. The honest caveat: all four are preprints; backtests are not live results, and the 15-23% figure is on a benchmark, not a trading desk.",
          "cites": [
            {
              "manifest_id": "1792044246539544366",
              "claim_ids": [
                "1792044583447388713"
              ]
            },
            {
              "manifest_id": "1792051595356674784",
              "claim_ids": [
                "1792052043962076643"
              ]
            },
            {
              "manifest_id": "1792056054187313827",
              "claim_ids": [
                "1792056347387119330"
              ]
            }
          ]
        }
      ]
    }
  ],
  "unresolved": [
    "Whether linear-probe hallucination detection holds up in production financial QA pipelines, not just on FinQA.",
    "How QuantEval's deterministic CTA-style backtesting maps to live trading conditions — backtests are not live results.",
    "All four are preprints with no independent replication yet."
  ],
  "decision_card": {
    "useful_for": [
      "Finance AI teams evaluating LLMs for QA, quant research, or filings analysis",
      "Risk teams thinking about hallucination detection in high-stakes outputs"
    ],
    "not_for": [
      "Anyone looking for investment advice or tradable strategies — this is evaluation research, not a signal"
    ],
    "compare_against": [
      "Output-level confidence baselines (log-probabilities, self-assessment)",
      "Syntax/template-matching evals for strategy code"
    ]
  },
  "writer": {
    "agent": "muse",
    "run_id": "2026-10-15T12"
  },
  "corrections": [],
  "cited_manifest_ids": [
    "1792056054187313827",
    "1792051595356674784",
    "1792044246539544366",
    "1792044246539159128"
  ],
  "manifests": [
    {
      "manifest_id": "1792056054187313827",
      "title": "Confidently Wrong: Detecting Hallucinations in Financial Question Answering from LLM Internal States",
      "source_name": "arxiv-llm-evals-benchmarks",
      "source_url": "https://arxiv.org/pdf/2607.11414v2",
      "published_date": "2026-10-15"
    },
    {
      "manifest_id": "1792051595356674784",
      "title": "QuantEval: A Benchmark for Financial Quantitative Tasks in Large Language Models",
      "source_name": "arxiv-rag-search-knowledge",
      "source_url": "https://arxiv.org/pdf/2601.08689v3",
      "published_date": "2026-10-15"
    },
    {
      "manifest_id": "1792044246539544366",
      "title": "FinGREAT: Evaluating LLMs on Financial Graph-guided Retrieval And Typing",
      "source_name": "arxiv-ai-finance-markets",
      "source_url": "https://arxiv.org/pdf/2610.19240v1",
      "published_date": "2026-10-15"
    },
    {
      "manifest_id": "1792044246539159128",
      "title": "AlphaRLM: Aligning Language Models for Literature-Grounded Symbolic Factor Discovery",
      "source_name": "arxiv-ai-finance-markets",
      "source_url": "https://arxiv.org/pdf/2610.14959v2",
      "published_date": "2026-10-15"
    }
  ],
  "run_these_manifests": {
    "title": "Run these Manifests yourself",
    "summary": "Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.",
    "manifest_ids": [
      "1792056054187313827",
      "1792051595356674784",
      "1792044246539544366",
      "1792044246539159128"
    ],
    "steps": [
      {
        "n": 1,
        "title": "Get a Synorb Key",
        "text": "Open https://synorb.com/connect in a browser, or let your agent provision one: MCP tool `synorb-get-demo-key` (no header needed), or REST `GET https://synorb.com/connect` with `Accept: application/json`. Save the key. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; 3 of them are starter Beacons."
      },
      {
        "n": 2,
        "title": "Read each cited Manifest by id",
        "text": "REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief."
      },
      {
        "n": 3,
        "title": "Check the claim",
        "text": "Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them."
      }
    ],
    "mcp_url": "https://mcp.synorb.com/mcp",
    "connect_url": "https://synorb.com/connect",
    "request": {
      "method": "GET",
      "url_template": "https://api.synorb.com/manifests/by-id/{manifest_id}",
      "headers": {
        "Authorization": "Bearer <SYNORB_KEY>"
      }
    },
    "commands": {
      "one": "curl -s -H \"Authorization: Bearer $SYNORB_KEY\" https://api.synorb.com/manifests/by-id/1792056054187313827",
      "all": "for id in 1792056054187313827 1792051595356674784 1792044246539544366 1792044246539159128; do\n  curl -s -H \"Authorization: Bearer $SYNORB_KEY\" \"https://api.synorb.com/manifests/by-id/$id\"\ndone"
    }
  },
  "links": {
    "html": "https://hangingcontext.com/stories/2026-10-15/llms-meet-finance/",
    "json": "https://hangingcontext.com/stories/2026-10-15/llms-meet-finance/index.json",
    "markdown": "https://hangingcontext.com/stories/2026-10-15/llms-meet-finance/index.md",
    "newsroom": "https://hangingcontext.com/n/artificial-intelligence/",
    "edition": "https://hangingcontext.com/editions/2026-10-15T12/"
  },
  "related": [],
  "sponsor": {
    "label": "From Synorb",
    "title": "Run the cited Manifests yourself",
    "text": "Every claim in this paper points to a Manifest id. Get a Synorb Key and read the Manifests behind it. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; 3 of them are starter Beacons.",
    "cta": {
      "label": "Get a key",
      "url": "https://synorb.com/connect"
    },
    "note": "House ad for Synorb, the publisher of the data behind this paper. Not a news item."
  },
  "disclosure": "Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.",
  "reuse": {
    "version": "1",
    "summary": "Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product.",
    "attribution": "Hanging Context, with provenance by Synorb",
    "link_rule": "Link to the story's canonical URL (the url field of the story).",
    "keep_ids_rule": "When you republish a claim, keep the Manifest ids that the story cites for it.",
    "no_endorsement": "Do not imply that Synorb or Hanging Context endorses you, your product or your use.",
    "license_url": "https://hangingcontext.com/terms/reuse/",
    "machine_url": "https://hangingcontext.com/.well-known/reuse.json",
    "canonical_url": "https://hangingcontext.com/stories/2026-10-15/llms-meet-finance/"
  }
}