{
  "schema": "hangingcontext.desk/1",
  "type": "story",
  "id": "1134",
  "slug": "finance-ai-reliability-outages",
  "edition": "2026-11-05T12",
  "url": "https://hangingcontext.com/stories/2026-11-05/finance-ai-reliability-outages/",
  "newsroom": "artificial-intelligence",
  "beat": "research",
  "newsroom_title": "Artificial Intelligence",
  "angle": "AI outages are now measured, and the trading agents that depend on them keep failing holdout",
  "headline": "AI outages are now a dataset: 5,868 of them, and trading agents keep failing holdout",
  "dek": "In preprints this week, an outage registry links AI downtime to fewer new trading executors, a bounded research loop shows LLM trading strategies failing holdout, and a framework argues benchmarks cannot measure AI reliability.",
  "published_at": "2026-11-05T12:29:45Z",
  "labels": {
    "trust": "reported",
    "trust_text": "Reported · 3 Manifests",
    "newsroom": "Artificial Intelligence"
  },
  "source_count": 3,
  "publishers": [
    "arxiv-ai-agents-tool-use",
    "arxiv-ai-finance-markets"
  ],
  "sources": [
    "arxiv-ai-agents-tool-use",
    "arxiv-ai-finance-markets"
  ],
  "distinct_source_count": 2,
  "key_facts": [
    {
      "text": "In a preprint, the number of active arbitrage addresses quadrupled between late 2024 and late 2026, while market execution in MEV remains highly concentrated despite the democratisation of trading code.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1793871826229351229",
          "claim_ids": [
            "1793872295007326246",
            "1793872295006018625"
          ]
        }
      ]
    },
    {
      "text": "In a preprint, LLM-generated trading strategies can pass discovery and validation with high performance while failing significantly in holdout testing.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1793871826228335353",
          "claim_ids": [
            "1793872289110708527"
          ]
        }
      ]
    },
    {
      "text": "In a preprint, average benchmark accuracy is insufficient to quantify the reliability of autonomous AI systems.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1793864792662675612",
          "claim_ids": [
            "1793865169792165972"
          ]
        }
      ]
    },
    {
      "text": "In a preprint, AI outages cause a measurable decline in the deployment of new trading executors.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1793871826229351229",
          "claim_ids": [
            "1793872295004003787"
          ]
        }
      ]
    }
  ],
  "images": [],
  "sections": [
    {
      "heading": "A registry of outages",
      "paragraphs": [
        {
          "markdown": "In a preprint posted this week, AIDA provides a registry of AI service outages, built with evidence from on-chain MEV data, as a dataset for evaluating the reliability of AI-driven trading infrastructure. Its headline finding is that outages matter: AI outages cause a measurable decline in the deployment of new trading executors, according to the preprint. When the AI service goes down, the trading code built on it stops being deployed. That is a reliability fact, not a capability fact, and it is the kind of fact a trading desk prices.",
          "cites": [
            {
              "manifest_id": "1793871826229351229",
              "claim_ids": [
                "1793872295004003787"
              ]
            }
          ]
        },
        {
          "markdown": "The preprint also describes a market that generative AI has not democratised. While generative AI tools have made creating trading contracts easier, market execution in MEV remains highly concentrated, according to the authors, and the number of active arbitrage addresses quadrupled between late 2024 and late 2026. The tools are widely available. The execution, and the private order flow behind it, are not. Infrastructure reliability and private order flow remain the primary determinants of competitive advantage, the preprint argues. The outage evidence is the basis for the first half of that claim.",
          "cites": [
            {
              "manifest_id": "1793871826229351229",
              "claim_ids": [
                "1793872295006018625",
                "1793872295007326246"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "The research loop that fools itself",
      "paragraphs": [
        {
          "markdown": "A second preprint shows why trading research done by agents needs a cage. The Autonomous Quant Researcher framework constrains LLM-driven trading research to prevent common pitfalls like look-ahead bias and overfitting, requiring proposals in strict declarative JSON and using a multi-gate acceptance process with immutable holdout data. It prevents model-written code execution by requiring proposals in strict declarative JSON, according to the preprint. Acceptance of trading strategies requires a conjunction of absolute gates including positive edge, net profit and loss, and stability metrics, rather than a single score.",
          "cites": [
            {
              "manifest_id": "1793871826228335353",
              "claim_ids": [
                "1793872289106143493",
                "1793872289107929777",
                "1793872289109898490"
              ]
            }
          ]
        },
        {
          "markdown": "The finding from two dozen campaigns is the warning. Agentic research loops in trading are prone to false discoveries due to overfitting, look-ahead bias, and multiple-comparisons artefacts, according to the preprint, and LLM-generated strategies can pass discovery and validation phases with high performance while failing significantly in holdout testing. A strategy that looks excellent in validation and fails in holdout is not a near-miss. It is the normal outcome of an unconstrained research loop. The framework's contribution is to make the failure visible before money is at risk.",
          "cites": [
            {
              "manifest_id": "1793871826228335353",
              "claim_ids": [
                "1793872289106143493",
                "1793872289110708527"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "Benchmarks cannot measure reliability",
      "paragraphs": [
        {
          "markdown": "A third preprint generalises the argument beyond finance. The authors argue that current benchmark-centric evaluation fails to capture the risks associated with agentic behaviours like tool use, memory, and autonomous reasoning, and that average benchmark accuracy is insufficient to quantify the reliability of autonomous AI systems. AI system failures can be classified into four levels, according to the preprint: component, operational-loop, agentic-conduct, and network or governance. Established reliability engineering principles such as failure mode analysis, accelerated testing, and failure reporting and corrective action systems are applicable to AI systems, the authors argue.",
          "cites": [
            {
              "manifest_id": "1793864792662675612",
              "claim_ids": [
                "1793865169792165972",
                "1793865169794009181",
                "1793865169796962181"
              ]
            }
          ]
        },
        {
          "markdown": "The authors propose a four-level diagnostic framework and the SMART framework to guide practitioners in building, monitoring, and validating AI systems beyond simple output correctness. The connection to the other two preprints is direct. AIDA counts the outages that benchmarks ignore. The quant research loop shows the false discoveries that validation scores hide. The reliability framework says the fix is to borrow from engineering disciplines that have measured failure for decades, rather than inventing a new benchmark and hoping it captures what matters.",
          "cites": [
            {
              "manifest_id": "1793864792662675612",
              "claim_ids": [
                "1793865169792165972",
                "1793865169794009181"
              ]
            }
          ]
        },
        {
          "markdown": "For a bank or a trading firm, the week's message is unglamorous and useful. The question is not whether the model can find a strategy. It is whether the service stays up, whether the strategy survives holdout, and whether the failure is classified and counted when it happens. The preprints are not peer-reviewed, and their numbers come from the authors' own experiments, but the direction they point is the same: reliability is now the scarce resource, and it is measurable if you count the right things.",
          "cites": [
            {
              "manifest_id": "1793871826229351229",
              "claim_ids": [
                "1793872295004003787"
              ]
            },
            {
              "manifest_id": "1793871826228335353",
              "claim_ids": [
                "1793872289110708527"
              ]
            },
            {
              "manifest_id": "1793864792662675612",
              "claim_ids": [
                "1793865169792165972"
              ]
            }
          ]
        }
      ]
    }
  ],
  "unresolved": [
    "Whether the 5,868-outage registry is complete, or misses outages without on-chain evidence",
    "How many LLM-generated strategies survive holdout under the bounded framework",
    "Whether banks adopt reliability engineering frameworks for AI, or keep benchmarking"
  ],
  "decision_card": {
    "useful_for": [
      "Bankers and traders evaluating AI-driven strategies",
      "Risk teams measuring AI system reliability"
    ],
    "not_for": [
      "Traders looking for a specific strategy",
      "Teams wanting peer-reviewed reliability guarantees"
    ],
    "compare_against": [
      "Benchmark accuracy as a reliability measure",
      "Unconstrained agent research loops"
    ]
  },
  "writer": {
    "agent": "muse",
    "run_id": "2026-11-05T12-ai"
  },
  "corrections": [],
  "cited_manifest_ids": [
    "1793871826229351229",
    "1793871826228335353",
    "1793864792662675612"
  ],
  "manifests": [
    {
      "manifest_id": "1793871826229351229",
      "title": "AIDA: An AI Outage Registry with Evidence from On-Chain MEV",
      "source_name": "arxiv-ai-finance-markets",
      "source_url": "https://arxiv.org/pdf/2611.01656v2",
      "published_date": "2026-11-05"
    },
    {
      "manifest_id": "1793871826228335353",
      "title": "Hard to Fool: A Bounded LLM Research Loop for Quantitative Finance, and What Two Dozen Campaigns Found",
      "source_name": "arxiv-ai-finance-markets",
      "source_url": "https://arxiv.org/pdf/2611.06607v1",
      "published_date": "2026-11-05"
    },
    {
      "manifest_id": "1793864792662675612",
      "title": "Reliability Engineering for AI Systems: Challenges, Methods, and Directions",
      "source_name": "arxiv-ai-agents-tool-use",
      "source_url": "https://arxiv.org/pdf/2609.35316v3",
      "published_date": "2026-11-05"
    }
  ],
  "run_these_manifests": {
    "title": "Run these Manifests yourself",
    "summary": "Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.",
    "manifest_ids": [
      "1793871826229351229",
      "1793871826228335353",
      "1793864792662675612"
    ],
    "steps": [
      {
        "n": 1,
        "title": "Get a Synorb Key",
        "text": "Click Generate Free Agent Key at https://synorb.com/connect/start, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{\"client\":\"your-agent-name\"}`. The key is the `api_key` field of the answer. Save the key. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons."
      },
      {
        "n": 2,
        "title": "Read each cited Manifest by id",
        "text": "REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief."
      },
      {
        "n": 3,
        "title": "Check the claim",
        "text": "Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them."
      }
    ],
    "mcp_url": "https://mcp.synorb.com/mcp",
    "connect_url": "https://synorb.com/connect/start",
    "request": {
      "method": "GET",
      "url_template": "https://api.synorb.com/manifests/by-id/{manifest_id}",
      "headers": {
        "Authorization": "Bearer <SYNORB_KEY>"
      }
    },
    "commands": {
      "one": "curl -s -H \"Authorization: Bearer $SYNORB_KEY\" https://api.synorb.com/manifests/by-id/1793871826229351229",
      "all": "for id in 1793871826229351229 1793871826228335353 1793864792662675612; do\n  curl -s -H \"Authorization: Bearer $SYNORB_KEY\" \"https://api.synorb.com/manifests/by-id/$id\"\ndone"
    }
  },
  "links": {
    "html": "https://hangingcontext.com/stories/2026-11-05/finance-ai-reliability-outages/",
    "json": "https://hangingcontext.com/stories/2026-11-05/finance-ai-reliability-outages/index.json",
    "markdown": "https://hangingcontext.com/stories/2026-11-05/finance-ai-reliability-outages/index.md",
    "newsroom": "https://hangingcontext.com/n/artificial-intelligence/",
    "edition": "https://hangingcontext.com/editions/2026-11-05T12/"
  },
  "related": [
    {
      "url": "https://hangingcontext.com/stories/2026-11-04/ai-power-permit-bottleneck/",
      "headline": "AI's bottleneck is power and permits, and it pushes revenue out",
      "slug": "ai-power-permit-bottleneck"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-10-31/agentic-wealth-synchronised-liquidation/",
      "headline": "When every wealth agent sells at once, the risk is synchrony, not a single bad model",
      "slug": "agentic-wealth-synchronised-liquidation"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-10-31/finance-agents-cant-calibrate/",
      "headline": "Finance agents can match analysts on the number and still miss the judgment",
      "slug": "finance-agents-cant-calibrate"
    }
  ],
  "sponsor": {
    "label": "From Synorb",
    "title": "Run the cited Manifests yourself",
    "text": "Every claim in this paper points to a Manifest id. Get a Synorb Key and read the Manifests behind it. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons.",
    "cta": {
      "label": "Get a key",
      "url": "https://synorb.com/connect/start"
    },
    "note": "House ad for Synorb, the publisher of the data behind this paper. Not a news item."
  },
  "disclosure": "Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.",
  "reuse": {
    "version": "1",
    "summary": "Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product.",
    "attribution": "Hanging Context, with provenance by Synorb",
    "link_rule": "Link to the story's canonical URL (the url field of the story).",
    "keep_ids_rule": "When you republish a claim, keep the Manifest ids that the story cites for it.",
    "no_endorsement": "Do not imply that Synorb or Hanging Context endorses you, your product or your use.",
    "license_url": "https://hangingcontext.com/terms/reuse/",
    "machine_url": "https://hangingcontext.com/.well-known/reuse.json",
    "canonical_url": "https://hangingcontext.com/stories/2026-11-05/finance-ai-reliability-outages/"
  }
}