Materials AI keeps failing the honest test. The fixes are in the data, not the model
Two preprints show general-purpose models flattered by their own evaluations, while small-data methods and better measurement tooling point to what actually works.
- General-purpose machine-learned interatomic potentials failed to give quantitative carbon dioxide isotherm predictions for metal-organic frameworks in a preprint evaluating a set of models. Self-reported m:…9926
- A random forest model scored R² 0.935 on random splits but −1.21 on own-paper holdouts for bismuth telluride Seebeck coefficient prediction. Self-reported m:…6243
- TabPFN predicted electrolyte conductivity with 5.56% mean absolute percentage error on the AMPERE dataset using 40 labelled samples. Self-reported m:…3376
- Bayesian optimisation guided by TabPFN cut experimental evaluations by 45.6% compared with random search. Self-reported m:…3376
- GITTcalc automates lithium diffusion-coefficient extraction from GITT data and checks whether the method's underlying assumptions hold. Self-reported m:…3375
The honest-test problem
Two preprints reach the same uncomfortable conclusion about machine learning in materials science: the models look strong until the test is honest. In one, researchers evaluated a set of universal machine-learned interatomic potentials on carbon dioxide adsorption in metal-organic frameworks, the porous materials used in carbon capture, and found the general-purpose models could not produce quantitative isotherm predictions. In the other, a team building a leakage-safe dataset for thermoelectric materials found that a random forest model predicting the Seebeck coefficient of bismuth telluride scored an R² of 0.935 on random splits — and minus 1.21 when tested on papers it had never seen. m:…9926 m:…6243
The pattern is the same in both cases: the evaluation flatters the model. The carbon-capture study traces the failure to errors in how the models describe the interaction between the framework and carbon dioxide, and between carbon dioxide molecules themselves, which sometimes cancel out by luck. The thermoelectric study shows that standard random splits inflate performance because near-duplicates leak across the train-test boundary. The lesson the two preprints share is that dataset composition matters more than model architecture — the carbon-capture authors say so directly — and that a benchmark that does not respect where data came from is not a benchmark. m:…9926 m:…6243
What survives an honest test
Not everything fails the honest test. A third preprint applies a tabular foundation model, TabPFN, to predicting electrolyte conductivity for batteries, working from as few as 40 labelled samples. On the AMPERE conductivity dataset the model reached a mean absolute percentage error of 5.56%, and when it guided Bayesian optimisation of electrolyte formulations it cut the number of experimental evaluations by 45.6% compared with random search — 38.2% fewer evaluations in the direct formulation search. The point is not the model. It is that small, well-chosen datasets can carry a real optimisation loop when the evaluation is set up to mean something. m:…3376
The fourth piece of the picture is about the data going in. GITTcalc, a new Python tool described in a preprint, automates the extraction of lithium diffusion coefficients from galvanostatic intermittent titration technique experiments, the standard way of measuring how fast lithium moves through a battery electrode. Manual analysis of these experiments is slow and error-prone, and the tool adds built-in checks for whether the experiment's underlying assumptions hold, plus a graphical interface and compatibility with OriginLab, the analysis software many battery labs already use. Reproducible numbers in, honest benchmarks out — the tooling and the testing are two halves of the same fix. m:…3375
Why this matters now
Materials informatics has spent several years importing methods from machine learning faster than it has imported machine learning's discipline about evaluation. That is starting to change, and the change is practical rather than philosophical: leakage-safe splits, source-labelled datasets, and small-data methods that say what they can do with 40 samples instead of implying they need far larger ones. For anyone funding or building in battery materials, carbon capture, or thermoelectrics, the question to ask of a model is no longer what it scores. It is what the test withheld. m:…6243 m:…3376
There is also a narrower lesson for teams choosing where to spend. The carbon-capture preprint argues that training-data composition beats architecture — which suggests money spent curating the right adsorption data buys more than money spent on a bigger model. The electrolyte work suggests the same from the other direction: a foundation model used as a cheap stand-in for experiments only earns its keep if the search it guides is measured against random search, not against nothing. The field is converging on a simple standard: show the holdout, name the leak, and count the experiments saved. m:…9926 m:…3376
What is still unresolved
- Whether universal interatomic potentials can be fixed by better training data alone, or whether carbon capture needs specialised models
- Whether TabPFN's small-data electrolyte results transfer to other electrolyte chemistries beyond the datasets tested
- Whether leakage-safe benchmarks like TEBench get adopted beyond thermoelectrics
Provenance by Synorb. Every claim links to its source. This story was written by an agent from the Manifests listed below. Read the methodology and limitations, or send a correction.
Free to reuse, including commercially. Credit “Hanging Context, with provenance by Synorb” and link to this story. Keep the cited Manifest ids. Reuse terms v1.