The forecasting audit: random splits lie, live benchmarks bite, and operators still win

Steel: the split is the lie

The sharpest numbers this week come from steel, where an LLM-orchestrated, physics-informed property prediction framework called SteelAgent was tested the way models actually meet the world. A gradient-boosting model scored far higher when test data was drawn at random from the same pool as training data than it did on held-out steel families it had never seen. Its uncertainty calibration collapsed with it: coverage of its 90% prediction intervals fell from 91.0% to 37.6%. Random splits, the standard way these models are evaluated, were systematically overstating how well the model generalizes. m:…0964

Even the physics did not save it cleanly. Adding physics-derived features helped in some evaluation populations and hurt in others, shifting R-squared by anywhere from plus 0.2534 to minus 0.358 depending on how the test set was structured. The lesson is not that physics-informed features are useless. It is that their value cannot be judged from a single split, and that a model which looks calibrated on familiar data can be badly miscalibrated on the next alloy family a lab actually cares about. m:…0964

Energy: live benchmarks bite

EnergyBench, a new benchmark for day-ahead energy forecasting, was built to stop exactly this kind of overfitting. It runs models live against a frozen information set at the market gate, evaluating eighteen time-series foundation models against baselines and grid operator forecasts across six European bidding zones. The findings are mixed in a useful way. Weather covariates reduced foundation model error by 37% to 51% across price, wind, solar, and load forecasts. Zero-shot foundation models matched or outperformed LightGBM models trained on the same weather covariates. m:…0954

But the human operators still win where it matters. Grid operator forecasts remained 16% to 22% more accurate than the best foundation models for wind and solar generation. The benchmark's value is that it shows this honestly: a live, gate-aligned test with no data leakage, where a foundation model can be genuinely good and still lose to the people who run the grid. For energy traders and grid planners, the takeaway is to use foundation models as a baseline, not as a replacement, and to demand live evaluation before believing a backtest. m:…0954

Volatility: a clean win, honestly measured

The week's cleanest forecasting result is in volatility. Neural Realized Measures replace fixed aggregation rules in volatility forecasting with a learned, context-dependent weighting scheme that optimizes for downstream forecasting performance, while keeping the structure of a weighted sum so the result stays interpretable and attributable. In a preprint, the method improved QLIKE in all 54 paired out-of-sample comparisons on six cryptocurrencies using five-minute data, with mean gains of 7.23% under one specification, 8.15% under a second, and 9.22% under a third. It also reduced error for latent integrated variance in Heston simulations across all evaluated primitive-regime combinations. m:…0960

A fourth preprint offers a method for the underlying dynamics. Diffusion-Driven State Space Models replace Gaussian transition distributions in deep state space models with diffusion models, allowing more expressive modeling of latent system dynamics, and outperformed state-of-the-art deep state space models on simulated time series with multimodal transitions. The authors frame it as a principled way to train autoencoders and diffusion models jointly on sequential data. Read together, the four papers describe one discipline: judge a forecast by how it was tested, not by how it sounds, and the tests that matter are the ones the model cannot game. m:…9650