The drug-screening pipeline that loses to random
A preprint audit finds a standard machine-learning screening pipeline produces candidates no better than random, while a simple positional statistic, a generative design workflow and a viral forecasting tool all produce results that survive contact with experiment.
- In a preprint, a standard machine-learning virtual screening pipeline produced COX-2 candidates showing no docking enrichment compared with random compounds. Self-reported m:…4730
- In the same preprint audit, 93% of the candidates generated by the Random Forest pipeline fell outside the model's applicability domain. Self-reported m:…4730
- In a preprint, a parameter-free positional lookup table outperformed biophysical models at predicting FGFR drug-resistance phenotypes, and the signal transferred across FGFR inhibitors from pemigatinib to futibatinib. Self-reported m:…4731
- In a preprint, REINVENT4-based generative design combined with similarity-guided screening identified 11 NCS-1 binders out of 12 tested compounds. Self-reported m:…4224
- The Arc Institute reports that its redesigned antibody X51 neutralized both the XFG variant and a forecasted variant, XFG-Next. Self-reported m:…2286
The audit that failed
A standard machine-learning pipeline for finding new drugs produced candidates no better than random, according to a preprint posted on ChemRxiv. The authors tested the pipeline on COX-2 as a case study and found that its chosen compounds showed no docking enrichment compared with random compounds drawn from the ZINC250k library. The result is a warning for a field that leans heavily on retrospective benchmarks: the pipeline looked strong on paper and failed in the prospective test. m:…4730
The gap between the benchmark and the outcome was stark. The model posted a Pearson correlation of 0.756 on retrospective tests, yet its predictions showed almost no relationship with docking scores in the prospective test, with a Spearman correlation of minus 0.086. Worse, 93 percent of the candidates the pipeline generated fell outside the model's applicability domain, with a mean maximum similarity to the training data of 0.243. The authors checked their yardstick first: the docking tool they used, AutoDock Vina, successfully distinguished genuine known COX-2 actives from random compounds, so the failure sits with the pipeline rather than with the test. m:…4730
A simple statistic beat the elaborate models
A second preprint, also on ChemRxiv, points the other way. Working on FGFR kinases, the authors found that a parameter-free positional lookup table outperformed established biophysical models, including AlphaMissense, FoldX, ThermoMPNN-D and AutoDock Vina, at predicting which mutations cause drug resistance. The signal was not tied to one drug: it transferred across FGFR inhibitors, from pemigatinib to futibatinib. It held across FGFR1, FGFR2 and FGFR3, though its predictive power weakened markedly for FGFR4. m:…4731
The contrast with the COX-2 audit is the point. One pipeline dressed up a familiar method and failed; a deliberately simple method, built on where in the protein a mutation sits, beat far more elaborate machinery. For teams choosing where to spend their modelling effort, the preprint suggests that the position of a mutation carries more signal than the physics-based scores layered on top of it. m:…4731 m:…4730
Design that survived contact with the lab
A third preprint offers a case where machine-led design did survive experiment. Using a generative design tool, REINVENT4, combined with similarity-guided screening, researchers identified 11 binders of the protein NCS-1 out of 12 compounds tested. The work produced the first small-molecule enhancers of the interaction between NCS-1 and the cannabinoid receptor CB1R, and high-resolution crystal structures confirmed that the new modulators engage the hydrophobic crevice of NCS-1. The difference from the COX-2 audit is that the claim rests on compounds that were made and measured, not on a benchmark score. m:…4224
Forecasting the virus instead of screening the library
The Arc Institute, meanwhile, is trying a different route altogether: forecasting viral evolution rather than screening compounds. Its tool antiGen is a machine-learning method trained on pandemic-scale phylogenetic trees, and it is a small model, with roughly half a million parameters. Trained on data through 2023, it predicted future mutations more accurately overall than the other methods it was compared with, and it also performed well on influenza, RSV and dengue, which have less sequencing data than COVID-19. m:…2286
Its companion tool, Germinal Multi, designs a single antibody against several variants of a target at once, and can redesign existing antibodies to keep them active against new variants. The researchers redesigned an antibody, X51, which neutralized both the XFG variant and a forecasted variant, XFG-Next. The same tool was used to design antibodies against EGFR mutations in colorectal cancer. The institute is open about the limit: antiGen cannot yet determine which circulating variants will be fitter than others. m:…2286
Taken together, the four reports describe a field sorting out what machine learning is actually good for. Generic screening pipelines can flatter themselves on benchmarks while producing nothing better than chance; simple, position-aware statistics and tools built on evolutionary history are producing results that survive contact with experiment. The test that mattered, in each case, was not the score on a held-out benchmark but what happened when the prediction met a real molecule, a real virus or a real crystal. m:…4730 m:…4731 m:…4224 m:…2286
What is still unresolved
- Whether the COX-2 audit generalises beyond one target and one pipeline, or whether other standard pipelines fail the same prospective test.
- Whether the FGFR positional statistic holds up on resistance data collected after the preprint, and why its power weakens for FGFR4.
- Whether Germinal Multi-designed antibodies hold their activity against variants that emerge after the design date.
Provenance by Synorb. Every claim links to its source. This story was written by an agent from the Manifests listed below. Read the methodology and limitations, or send a correction.
Free to reuse, including commercially. Credit “Hanging Context, with provenance by Synorb” and link to this story. Keep the cited Manifest ids. Reuse terms v1.