AI outages are now a dataset: 5,868 of them, and trading agents keep failing holdout

A registry of outages

In a preprint posted this week, AIDA provides a registry of AI service outages, built with evidence from on-chain MEV data, as a dataset for evaluating the reliability of AI-driven trading infrastructure. Its headline finding is that outages matter: AI outages cause a measurable decline in the deployment of new trading executors, according to the preprint. When the AI service goes down, the trading code built on it stops being deployed. That is a reliability fact, not a capability fact, and it is the kind of fact a trading desk prices. m:…1229

The preprint also describes a market that generative AI has not democratised. While generative AI tools have made creating trading contracts easier, market execution in MEV remains highly concentrated, according to the authors, and the number of active arbitrage addresses quadrupled between late 2024 and late 2026. The tools are widely available. The execution, and the private order flow behind it, are not. Infrastructure reliability and private order flow remain the primary determinants of competitive advantage, the preprint argues. The outage evidence is the basis for the first half of that claim. m:…1229

The research loop that fools itself

A second preprint shows why trading research done by agents needs a cage. The Autonomous Quant Researcher framework constrains LLM-driven trading research to prevent common pitfalls like look-ahead bias and overfitting, requiring proposals in strict declarative JSON and using a multi-gate acceptance process with immutable holdout data. It prevents model-written code execution by requiring proposals in strict declarative JSON, according to the preprint. Acceptance of trading strategies requires a conjunction of absolute gates including positive edge, net profit and loss, and stability metrics, rather than a single score. m:…5353

The finding from two dozen campaigns is the warning. Agentic research loops in trading are prone to false discoveries due to overfitting, look-ahead bias, and multiple-comparisons artefacts, according to the preprint, and LLM-generated strategies can pass discovery and validation phases with high performance while failing significantly in holdout testing. A strategy that looks excellent in validation and fails in holdout is not a near-miss. It is the normal outcome of an unconstrained research loop. The framework's contribution is to make the failure visible before money is at risk. m:…5353

Benchmarks cannot measure reliability

A third preprint generalises the argument beyond finance. The authors argue that current benchmark-centric evaluation fails to capture the risks associated with agentic behaviours like tool use, memory, and autonomous reasoning, and that average benchmark accuracy is insufficient to quantify the reliability of autonomous AI systems. AI system failures can be classified into four levels, according to the preprint: component, operational-loop, agentic-conduct, and network or governance. Established reliability engineering principles such as failure mode analysis, accelerated testing, and failure reporting and corrective action systems are applicable to AI systems, the authors argue. m:…5612

The authors propose a four-level diagnostic framework and the SMART framework to guide practitioners in building, monitoring, and validating AI systems beyond simple output correctness. The connection to the other two preprints is direct. AIDA counts the outages that benchmarks ignore. The quant research loop shows the false discoveries that validation scores hide. The reliability framework says the fix is to borrow from engineering disciplines that have measured failure for decades, rather than inventing a new benchmark and hoping it captures what matters. m:…5612

For a bank or a trading firm, the week's message is unglamorous and useful. The question is not whether the model can find a strategy. It is whether the service stays up, whether the strategy survives holdout, and whether the failure is classified and counted when it happens. The preprints are not peer-reviewed, and their numbers come from the authors' own experiments, but the direction they point is the same: reliability is now the scarce resource, and it is measurable if you count the right things. m:…1229 m:…5353 m:…5612