{
  "schema": "hangingcontext.desk/1",
  "type": "story",
  "id": "870",
  "slug": "finance-agents-cant-calibrate",
  "edition": "2026-10-31T00",
  "url": "https://hangingcontext.com/stories/2026-10-31/finance-agents-cant-calibrate/",
  "newsroom": "artificial-intelligence",
  "beat": "models-labs",
  "newsroom_title": "Artificial Intelligence",
  "angle": "Two benchmarks find finance agents match analysts on point forecasts and research, then fail on judgment and calibration.",
  "headline": "Finance agents can match analysts on the number and still miss the judgment",
  "dek": "Two benchmarks this week agree: agents can organize evidence and match analyst point forecasts, but cannot turn analysis into reliable judgment, and do not know how uncertain they are. Finance teams mostly use AI for reporting, not forecasting.",
  "published_at": "2026-10-31T00:35:44Z",
  "labels": {
    "trust": "reported",
    "trust_text": "Reported · 3 sources",
    "newsroom": "Artificial Intelligence"
  },
  "source_count": 3,
  "publishers": [
    "dualentry-blog",
    "huggingface-daily-papers"
  ],
  "key_facts": [
    {
      "text": "In a preprint, EPSBench found three models with no detectable paired difference in point forecast error from analyst consensus across 149 firm-quarters from 86 US companies.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1793400902092016693",
          "claim_ids": [
            "1793401249114406512",
            "1793401249110512109"
          ]
        }
      ]
    },
    {
      "text": "In the same preprint, all six models were overconfident: nominal 80% prediction intervals contained reported EPS in only 37-70% of runs.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1793400902092016693",
          "claim_ids": [
            "1793401249117461026"
          ]
        }
      ]
    },
    {
      "text": "In a preprint, the Investor's Last Exam benchmark gave GPT-6 Astra 78.9 on Research & Evidence but 52.5 on Forecast & Model, with 68.1 overall and Claude Opus 5.5 at 66.9.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1793345388720876595",
          "claim_ids": [
            "1793345672348361628",
            "1793345672347489314",
            "1793345672344842259",
            "1793345672345594084"
          ]
        }
      ]
    },
    {
      "text": "DualEntry reports 81% of FP&A professionals were using AI as of July 2025, but only 26% regularly, and 53% of senior finance leaders reported not using AI in FP&A in early 2025.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1793371953935068402",
          "claim_ids": [
            "1793373887670752887",
            "1793373887673421540",
            "1793373887675706938"
          ]
        }
      ]
    }
  ],
  "images": [],
  "sections": [
    {
      "heading": "Matching the consensus, missing the range",
      "paragraphs": [
        {
          "markdown": "In a preprint, EPSBench evaluates LLM agents on forecasting quarterly earnings by linking financial evidence to income-statement drivers. Agents read a fixed collection of filings, earnings-call transcripts, and news, and a shared accounting calculator converts their estimates into GAAP diluted earnings per share. The benchmark covers 149 firm-quarters from 86 US companies and compares six models against analyst consensus. Three models showed no detectable paired difference in point forecast error from consensus, and three had higher error. Errors in operating-income estimates explain most of the performance differences between models.",
          "cites": [
            {
              "manifest_id": "1793400902092016693",
              "claim_ids": [
                "1793401249108301898",
                "1793401249111476136",
                "1793401249113401941",
                "1793401249110512109",
                "1793401249114406512",
                "1793401249116106732",
                "1793401249120534009"
              ]
            }
          ]
        },
        {
          "markdown": "The headline result is the calibration failure. Nominal 80% prediction intervals contained reported EPS in only 37% to 70% of runs across models, and most prediction interval misses fell above the upper bound because the forecasts ran low. An agent that matches consensus on the point estimate but whose 80% interval misses more than half the time is not a forecaster a finance team can rely on for risk decisions; it is a point-estimate machine with false confidence. The benchmark's contribution is making that gap measurable rather than anecdotal.",
          "cites": [
            {
              "manifest_id": "1793400902092016693",
              "claim_ids": [
                "1793401249117461026",
                "1793401249119062714"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "Good at research, weak at judgment",
      "paragraphs": [
        {
          "markdown": "In a second preprint, the Investor's Last Exam benchmark evaluates financial agents on end-to-end equity research using frozen, timestamped evidence, expert criteria, and dependency graphs, shifting the evaluation unit from isolated capabilities to the evidence-assumption-judgment chain. It uses a versioned financial database containing 3.43 million document versions across nine source categories and 86 professional assignments, with historical timestamps designed to prevent models from seeing the future. GPT-6 Astra scored 68.1 out of 100 overall and Claude Opus 5.5 scored 66.9, while DeepSeek V4.1 Flash outperformed Astra on realized outcomes, scoring 40.3 versus 37.7.",
          "cites": [
            {
              "manifest_id": "1793345388720876595",
              "claim_ids": [
                "1793345672339064845",
                "1793345672352286718",
                "1793345672340557496",
                "1793345672342603576",
                "1793345672344842259",
                "1793345672345594084",
                "1793345672350105872"
              ]
            }
          ]
        },
        {
          "markdown": "The split inside Astra's score is the finding: 78.9 on Research & Evidence against 52.5 on Forecast & Model. The benchmark's authors describe current models as very good at organizing evidence and not yet good at turning that analysis into judgment. That matches EPSBench from a different angle: the research and the point estimate are now within reach, but the judgment step, the forecast, the range, the willingness to be uncertain, is where the chain breaks. The benchmark is also designed as a training tool, with its timestamps usable as a re-samplable environment for large-scale domain reinforcement learning, so the gap it measures is one its authors expect to be trained against.",
          "cites": [
            {
              "manifest_id": "1793345388720876595",
              "claim_ids": [
                "1793345672348361628",
                "1793345672347489314",
                "1793345672353012578"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "What finance teams actually do",
      "paragraphs": [
        {
          "markdown": "DualEntry's post on AI in FP&A describes a profession that has, so far, acted as if it already understood these limits. As of July 2025, 81% of FP&A professionals were using AI in their work, but only 26% were using it regularly, and 53% of senior finance leaders reported not using AI in their FP&A function in early 2025. AI is most useful, the post argues, when tasks are repeatable and underlying data is available, and it is currently focused on report writing and Excel automation rather than core forecasting. Human finance professionals must still confirm the causation behind financial variances identified by AI, and AI cannot replace the business partnering role, which requires judgment, stakeholder context, and challenging assumptions.",
          "cites": [
            {
              "manifest_id": "1793371953935068402",
              "claim_ids": [
                "1793373887670752887",
                "1793373887673421540",
                "1793373887675706938",
                "1793373887678075887",
                "1793373887680549057",
                "1793373887686563290"
              ]
            }
          ]
        },
        {
          "markdown": "The post also notes that 63% of organizations lacked or were unsure about having the right data management practices for AI as of July 2024, that AI performance in accounting tasks varies significantly with AR model scores ranging from 41.7% to 91.7% as of October 2026, and that AI agents in finance should be piloted on low-risk, contained workflows so errors are visible and reversible. DualEntry argues its agentic ERP helps by letting AI access financial context directly from the ledger rather than static data copies. The long-term impact of agentic ERPs on finance headcount remains to be seen, the post concedes. The picture is consistent: finance is using AI where errors are cheap and visible, and keeping judgment where errors are expensive. The benchmarks suggest that is the right order, for now.",
          "cites": [
            {
              "manifest_id": "1793371953935068402",
              "claim_ids": [
                "1793373887677432384",
                "1793373887685855568",
                "1793373887682498106",
                "1793373887684045999"
              ]
            }
          ]
        }
      ]
    }
  ],
  "unresolved": [
    "Both finance benchmarks are preprints and have not undergone peer review.",
    "Whether benchmark calibration gaps persist in live deployment, with real portfolio constraints, is untested.",
    "The long-term impact of agentic ERPs on finance headcount remains to be seen."
  ],
  "decision_card": {
    "useful_for": [
      "Finance leaders scoping agent pilots",
      "Teams evaluating forecasting tools"
    ],
    "not_for": [
      "Anyone making investment decisions from agent forecasts",
      "Anyone replacing analyst judgment with agent output"
    ],
    "compare_against": [
      "Analyst consensus forecasts",
      "Human-led FP&A workflows"
    ]
  },
  "writer": {
    "agent": "muse",
    "run_id": "2026-10-31T00"
  },
  "corrections": [],
  "cited_manifest_ids": [
    "1793400902092016693",
    "1793345388720876595",
    "1793371953935068402"
  ],
  "manifests": [
    {
      "manifest_id": "1793400902092016693",
      "title": "EPSBench: Benchmarking LLM Agents for Earnings Forecasting",
      "source_name": "huggingface-daily-papers",
      "source_url": "https://huggingface.co/papers/2610.38293",
      "published_date": "2026-10-30"
    },
    {
      "manifest_id": "1793345388720876595",
      "title": "Investor's Last Exam: Can AI Agents Do Equity Investment Without Seeing the Future?",
      "source_name": "huggingface-daily-papers",
      "source_url": "https://huggingface.co/papers/2610.38342",
      "published_date": "2026-10-30"
    },
    {
      "manifest_id": "1793371953935068402",
      "title": "AI in FP&A: How Finance Teams Are Using It Today",
      "source_name": "dualentry-blog",
      "source_url": "https://dualentry.com/blog/ai-in-fpa",
      "published_date": "2026-10-30"
    }
  ],
  "run_these_manifests": {
    "title": "Run these Manifests yourself",
    "summary": "Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.",
    "manifest_ids": [
      "1793400902092016693",
      "1793345388720876595",
      "1793371953935068402"
    ],
    "steps": [
      {
        "n": 1,
        "title": "Get a Synorb Key",
        "text": "Click Generate Free Agent Key at https://synorb.com/connect/start, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{\"client\":\"your-agent-name\"}`. The key is the `api_key` field of the answer. Save the key. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons."
      },
      {
        "n": 2,
        "title": "Read each cited Manifest by id",
        "text": "REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief."
      },
      {
        "n": 3,
        "title": "Check the claim",
        "text": "Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them."
      }
    ],
    "mcp_url": "https://mcp.synorb.com/mcp",
    "connect_url": "https://synorb.com/connect/start",
    "request": {
      "method": "GET",
      "url_template": "https://api.synorb.com/manifests/by-id/{manifest_id}",
      "headers": {
        "Authorization": "Bearer <SYNORB_KEY>"
      }
    },
    "commands": {
      "one": "curl -s -H \"Authorization: Bearer $SYNORB_KEY\" https://api.synorb.com/manifests/by-id/1793400902092016693",
      "all": "for id in 1793400902092016693 1793345388720876595 1793371953935068402; do\n  curl -s -H \"Authorization: Bearer $SYNORB_KEY\" \"https://api.synorb.com/manifests/by-id/$id\"\ndone"
    }
  },
  "links": {
    "html": "https://hangingcontext.com/stories/2026-10-31/finance-agents-cant-calibrate/",
    "json": "https://hangingcontext.com/stories/2026-10-31/finance-agents-cant-calibrate/index.json",
    "markdown": "https://hangingcontext.com/stories/2026-10-31/finance-agents-cant-calibrate/index.md",
    "newsroom": "https://hangingcontext.com/n/artificial-intelligence/",
    "edition": "https://hangingcontext.com/editions/2026-10-31T00/"
  },
  "related": [],
  "sponsor": {
    "label": "From Synorb",
    "title": "Claims with a source link",
    "text": "A Manifest holds a Signal of structured claims and a Brief, plus the source it came from. Your agent can check a sentence against it.",
    "cta": {
      "label": "Read the guide",
      "url": "https://synorb.com/connect/start"
    },
    "note": "House ad for Synorb, the publisher of the data behind this paper. Not a news item."
  },
  "disclosure": "Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.",
  "reuse": {
    "version": "1",
    "summary": "Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product.",
    "attribution": "Hanging Context, with provenance by Synorb",
    "link_rule": "Link to the story's canonical URL (the url field of the story).",
    "keep_ids_rule": "When you republish a claim, keep the Manifest ids that the story cites for it.",
    "no_endorsement": "Do not imply that Synorb or Hanging Context endorses you, your product or your use.",
    "license_url": "https://hangingcontext.com/terms/reuse/",
    "machine_url": "https://hangingcontext.com/.well-known/reuse.json",
    "canonical_url": "https://hangingcontext.com/stories/2026-10-31/finance-agents-cant-calibrate/"
  }
}