Chemistry AI keeps passing the wrong test, and the gap is now measured
Three preprints show random splits flatter chemistry models by wide margins. A fourth shows what honest evaluation looks like — and the old baseline nearly wins.
- Reliability gates calibrated on held-out chemistry reach 0.85-0.87 pooled precision on Buchwald-Hartwig yield tests with unseen additives; random-fold calibration manages only 0.59-0.75 and misses the 0.8 target. Self-reported m:…6614
- Enantioselectivity models drop from 66.1% accuracy on random splits to 52.9% under leave-one-paper-out testing. Self-reported m:…4095
- Publication-block splitting raises PLGA release prediction error to an MAE of 0.172 from 0.136 under formulation-random splitting, and publication-max calibration covers 88% of endpoints versus 68% for pooled-point calibration. Self-reported m:…3120
- On a scaffold-disjoint benchmark, tuned KNN-20 reaches Hits@10 of 0.9335 against 0.9355 for a multitask MLP ensemble; a 40% KNN / 60% MLP hybrid reaches 0.9426. Self-reported m:…9862
The gap, measured three ways
Three new preprints in chemistry machine learning point at the same problem from three directions: the way these models are usually tested makes them look better than they are. The papers cover reaction-yield prediction, enantioselectivity, and drug-release modelling, and in each case the headline number falls when the test set is built from chemistry the model has not seen — a new additive, a new paper, a new publication. m:…6614 m:…4095 m:…3120
The clearest case is a test of accept-or-abstain reliability gates for molecular property and reaction-yield models. When the gates were calibrated on held-out chemistry, they hit a pooled precision of 0.85 to 0.87 on Buchwald-Hartwig reaction-yield tests with unseen additives. When the same gates were calibrated on random folds — the standard split — precision fell to between 0.59 and 0.75, below the 0.8 target the authors set. Conformal intervals calibrated on a random fold lost coverage under distribution shift, dropping to 0.64. m:…6614
A second paper re-ran enantioselectivity prediction on polysaccharide chiral stationary phases under a leave-one-paper-out protocol, and accuracy fell from 66.1% under random splits to 52.9% — close to chance. Between-paper variance accounted for 38.5% of total variance and could not be predicted from compositional features, and in a single-laboratory grid 76.95% of variance came from molecule-by-condition interactions and noise the authors could not identify. m:…4095
A third paper, on predicting drug-release endpoints from PLGA formulations, found the same pattern in a different corner of the field. Splitting the data by publication instead of by formulation raised the error of an Extra-trees model from a mean absolute error of 0.136 to 0.172, and calibration by publication reached 88% empirical coverage of release endpoints against 68% for pooled-point calibration. The author is direct about what remains unknown: whether models trained on literature-mined data generalize at all is still unestablished. m:…3120
What honest evaluation looks like
The fourth paper shows what the honest version of this work looks like, and the result is less dramatic than the usual story. On a scaffold-disjoint benchmark, a tuned classical similarity search — KNN-20 — scored a Hits@10 of 0.9335, essentially level with a multitask neural network ensemble at 0.9355. Blending the two, 40% classical and 60% neural, reached 0.9426 and beat both. The neural model did not lose; it just did not win by much, and the win came from combining representations rather than replacing one with the other. m:…9862
Taken together, the four papers describe a field where the evaluation protocol is doing more work than the model architecture. The pattern is consistent with a simple explanation: random splits let near-duplicates of the test set into training, so the model is partly recalling rather than generalizing. When the split respects the boundaries that matter in practice — new chemistry, new papers, new publications — the recall advantage disappears and the real performance shows. m:…6614 m:…4095 m:…3120 m:…9862
What changes for teams that buy or build these models
For teams buying or building chemistry models, the practical read is straightforward. Ask how the vendor split the data before asking how accurate the model is. A model that reports precision above 0.8 on a random fold has not shown it can hold that precision on new chemistry — the gates paper shows it falling to between 0.59 and 0.75. And a neural model that beats a tuned classical baseline by a fraction of a point on a rigorous split may not be worth the complexity; the hybrid result suggests the cheapest improvement is often to combine the two rather than to replace one with the other. m:…6614 m:…9862
The deeper implication is about trust. These models are being built into drug-discovery workflows where a wrong prediction costs a synthesis run, not just a benchmark point. Until held-out chemistry becomes the default test — the way the gates paper and the PLGA paper both argue it should be — the numbers on the slide will keep overstating what the model can do on the next molecule. m:…6614 m:…3120
What is still unresolved
- Whether reliability gates calibrated on held-out chemistry hold their precision on production-scale reaction data beyond the Buchwald-Hartwig tests.
- Whether models trained on literature-mined PLGA data can support prospective prediction at all — the paper says this is unestablished.
- What the hybrid KNN-MLP approach costs in compute and maintenance versus the small gain over either model alone.
Provenance by Synorb. Every claim links to its source. This story was written by an agent from the Manifests listed below. Read the methodology and limitations, or send a correction.
Free to reuse, including commercially. Credit “Hanging Context, with provenance by Synorb” and link to this story. Keep the cited Manifest ids. Reuse terms v1.