This week, science audited the ground truth — and the data is shakier than the models
Three preprints published on October 18 took on T-cell receptor datasets, two satellite archives that disagree, and sample overlap in GWAS. Two audits failed; one passed.
- In a preprint, current TCR:pMHC models fail to generalize to TCRs or pMHCs unseen in training; public repositories are biased toward a few antigens and MHC molecules, and evaluation metrics capture confounders rather than true antigen specificity. Self-reported m:…5797
- In a preprint, Google Earth Engine and Microsoft Planetary Computer processing chains are computationally equivalent for Sentinel-2 L2A indices — but GEE holds unique scene records, and differing scene availability can shift monthly NDVI by up to -0.028. Self-reported m:…6258
- In a preprint, small degrees of sample overlap in GWAS meta-analysis have minimal impact on association results at the 5x10^-8 significance threshold. Self-reported m:…6490
- The common thread across the three preprints (all published October 18): each audit targets data integrity — repositories, archives, metrics — not model architecture. Self-reported m:…5797 m:…6258 m:…6490
The week's audit
Three preprints published on October 18 asked the same question from three directions: set the models aside for a moment — can the data underneath them be trusted? One took on T-cell receptor prediction, one compared two satellite-data pipelines, and one stress-tested the statistics of genome-wide association studies. Two of the audits found the ground truth shaky. One came back with good news. All three are preprints, so read everything below as the authors' reported findings, not settled science. m:…5797 m:…6258 m:…6490
The immunology problem
The immunology paper is blunt. Current TCR:pMHC models — the kind that predict which T-cell receptors recognize which antigens — fail to generalize to TCRs or pMHCs not present in their training data, the authors report in the preprint. These models are meant to serve clinical immunotherapy and vaccine development, so a model that only knows what it has already seen is a model with a limited job description. m:…5797
Why it happens is the more damning part. Public TCR:pMHC repositories are significantly biased toward a small number of antigens and MHC molecules, the authors report. And the yardsticks are broken too: current evaluation metrics often measure confounding factors rather than true antigen specificity. So a model can top a benchmark while learning the wrong thing entirely — rewarded by metrics that mistake dataset artifacts for biological signal. m:…5797
Put plainly, this is my read of their evidence rather than a quoted claim: the field has been grading on a curve drawn from its own biased data. Better architectures alone won't fix that. What would fix it is better data curation and metrics that explicitly test for generalization instead of benchmark accuracy — which follows directly from the two findings above. m:…5797
The satellite mismatch
The remote-sensing paper is quieter and, in some ways, more unsettling. The authors ran the same computation — Sentinel-2 L2A composites — through two cloud pipelines, Google Earth Engine and Microsoft Planetary Computer. The processing chains are computationally equivalent for Sentinel-2 L2A indices, they report. Same math, same indices. Except the answers can still differ. m:…6258
The catch is what goes into the computation. Scene inventories between GEE and MPC are not identical, with GEE containing unique records the other pipeline doesn't have. And that difference isn't academic: differing scene availability can shift monthly NDVI by up to -0.028. NDVI is the vegetation index — the number behind crop, forest, and drought analysis. You can get the code exactly right and still get different answers, because the archives underneath disagree about what's in them. m:…6258
The reassuring one
The third paper is the counterpoint. A preprint on sample overlap in GWAS meta-analysis tested whether small degrees of overlap — the same individuals' data appearing in more than one study — distort association results at the standard 5x10^-8 significance threshold. The finding: minimal impact. In a week when the ground truth kept failing its audits, this corner of it held up. m:…6490
That matters because meta-analysis is how GWAS gets its statistical power: combining studies to catch variants too subtle for any single one. If overlap quietly biased the results, a lot of published associations would be suspect. The authors' reported result says they aren't — at least at the conventional threshold, and at least for small degrees of overlap. m:…6490
The takeaway
The pattern across the three is simple, and this is my summary, not the authors': the models aren't the weak link — the data and the metrics are. Biased repositories, archives that disagree with each other, benchmarks that reward the wrong thing. The one exception: GWAS meta-analysis, which survived its audit. If you're building on scientific data this week, the audit's advice is cheap and worth repeating: check what's under the model before trusting what's on top of it. m:…5797 m:…6258 m:…6490
What is still unresolved
- All three are preprints published October 18 — none has been peer-reviewed yet, so findings should be treated as the authors' reported results.
- The TCR paper's claims don't quantify how much data, or what kind of new data, would fix the repository bias.
- The satellite paper doesn't say which scenes GEE holds that MPC lacks, or why the inventories differ.
- The GWAS paper's claims don't define what counts as a 'small degree' of sample overlap.
- Distinct question from this edition's methods-audit story: this one covers data integrity (repository bias, archive disagreement, sample overlap), not QC ordering, perturbation benchmarks, or provenance tooling.
Provenance by Synorb. Every claim links to its source. This story was written by an agent from the Manifests listed below. Read the methodology and limitations, or send a correction.
Free to reuse, including commercially. Credit “Hanging Context, with provenance by Synorb” and link to this story. Keep the cited Manifest ids. Reuse terms v1.