Science is stress-testing its own AI

Open baselines bite back

Three studies published October 19-20 turn machine learning's evaluation machinery on itself: open baselines for language models, a bake-off for viral prediction tools, and an autopsy of a broken train-test split. The first is open-sci-ref-0.02, a suite of open reference baselines for reproducible language model and dataset comparison published October 20 as a preprint — and it is already embarrassing models. Its headline finding: EuroLLM-1.7B underperforms the FineWeb-Edu baseline despite 13 times more training compute. Nemotron-CC-HQ emerges as the strongest reference dataset in the suite. The message is uncomfortable for scale maximalists: more compute on worse data loses to less compute on better data. m:…1569

The methodological prescription: data ranking should be evaluated through multi-scale reference trajectories rather than single proxy runs. One proxy run is an anecdote — reasoning: it confounds what the data contributes with what the training recipe contributes, and you cannot tell which one you are measuring. Multi-scale trajectories separate the two. That is the difference between a benchmark and a leaderboard, and the field has been running too many leaderboards. m:…1569

The broader pattern — reasoning — is that evaluation is catching up to capability. Open baselines, tool bake-offs, split autopsies: each one makes a published claim harder to fake and easier to check. A field that audits its own measurement instruments is a field growing up, and this week science did it three times at once. m:…1569

Benchmarking the viral dark matter

Metagenomics has a dark matter problem: most viral sequences in low-abundance samples match nothing known. A preprint benchmarking viral prediction tools across divergent low-abundance metagenomes finds DeepVirfinder and DIAMOND achieved the highest recall — 67% and 58% respectively — on the RefSeq artificial contigs dataset. Four two-tool combinations (DeepVirfinder paired with DeepMicroclass, ViraLM, geNomad, or DIAMOND), each with Kraken2X filtering, achieved the highest sensitivity and F1 scores. Kraken2X filtering, meanwhile, proved effective at removing false-positive contigs classified as Eukaryota, Bacteria, or Archaea — the contamination that drowns real viral signal in low-abundance samples. m:…3417

The practical takeaway — reasoning: no single tool wins; pairs win, and filtering wins. For labs dredging low-abundance samples for novel viruses, the benchmark is a shopping list, not a leaderboard: pick a two-tool combination, filter aggressively, and stop pretending one detector does the job. m:…3417

The split that wasn't

The sharpest audit targets machine learning for materials science. In the hMOF dataset, 74.7% of coarse compositions occur in a single topology — chemistry and topology are entangled. Removing the nbo topology in the pcu-to-nbo pair degrades MAE by 70.0%, and the paper's conclusion is blunt: topology hold-out is not a clean test of topological transferability, because the grouped split silently removes the chemistry entangled with its grouping variable. The model was not learning topology; it was learning which chemistry came with which topology. m:…6527

This is the kind of failure that flatters — reasoning: the hold-out looks rigorous, the metric looks clean, and the conclusion is wrong. Any field using grouped splits to claim generalization should be re-reading its splits this week, because the confounder is always the variable you grouped by and forgot to check. m:…6527

The fix is not mysterious — reasoning from the entanglement finding: splits must be designed against the confounder, not just the label. Stratify by chemistry first, then hold out topology — or accept that the test measures something else and say so plainly. Honest measurement beats impressive measurement, and this paper shows exactly what impressive-but-dishonest looks like. m:…6527