{
  "schema": "hangingcontext.desk/1",
  "type": "story",
  "id": "250",
  "slug": "ground-truth-audit",
  "edition": "2026-10-19T00",
  "url": "https://hangingcontext.com/stories/2026-10-19/ground-truth-audit/",
  "newsroom": "research-science",
  "beat": null,
  "newsroom_title": "Research & Science",
  "angle": "This week's preprints audited the data under the models: biased immunology sets, disagreeing satellite catalogues, one reassuring GWAS.",
  "headline": "This week, science audited the ground truth — and the data is shakier than the models",
  "dek": "Three preprints published on October 18 took on T-cell receptor datasets, two satellite archives that disagree, and sample overlap in GWAS. Two audits failed; one passed.",
  "published_at": "2026-10-19T01:03:16Z",
  "labels": {
    "trust": "confirmed",
    "trust_text": "Confirmed · 3 sources",
    "newsroom": "Research & Science"
  },
  "source_count": 3,
  "publishers": [
    "biorxiv-cancer-immunology-translation",
    "biorxiv-genomics-omics-platforms",
    "eartharxiv-ocean-cryosphere-sea-level"
  ],
  "key_facts": [
    {
      "text": "In a preprint, current TCR:pMHC models fail to generalize to TCRs or pMHCs unseen in training; public repositories are biased toward a few antigens and MHC molecules, and evaluation metrics capture confounders rather than true antigen specificity.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1792359031773665797",
          "claim_ids": [
            "1792359270094899663",
            "1792359270097327067",
            "1792359270099254934"
          ]
        }
      ]
    },
    {
      "text": "In a preprint, Google Earth Engine and Microsoft Planetary Computer processing chains are computationally equivalent for Sentinel-2 L2A indices — but GEE holds unique scene records, and differing scene availability can shift monthly NDVI by up to -0.028.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1792360248362486258",
          "claim_ids": [
            "1792360778467755135",
            "1792360778469271888",
            "1792360778471879801"
          ]
        }
      ]
    },
    {
      "text": "In a preprint, small degrees of sample overlap in GWAS meta-analysis have minimal impact on association results at the 5x10^-8 significance threshold.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1792314879401536490",
          "claim_ids": [
            "1792315411901680521"
          ]
        }
      ]
    },
    {
      "text": "The common thread across the three preprints (all published October 18): each audit targets data integrity — repositories, archives, metrics — not model architecture.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1792359031773665797",
          "claim_ids": [
            "1792359270094899663"
          ]
        },
        {
          "manifest_id": "1792360248362486258",
          "claim_ids": [
            "1792360778471879801"
          ]
        },
        {
          "manifest_id": "1792314879401536490",
          "claim_ids": [
            "1792315411901680521"
          ]
        }
      ]
    }
  ],
  "images": [],
  "sections": [
    {
      "heading": "The week's audit",
      "paragraphs": [
        {
          "markdown": "Three preprints published on October 18 asked the same question from three directions: set the models aside for a moment — can the data underneath them be trusted? One took on T-cell receptor prediction, one compared two satellite-data pipelines, and one stress-tested the statistics of genome-wide association studies. Two of the audits found the ground truth shaky. One came back with good news. All three are preprints, so read everything below as the authors' reported findings, not settled science.",
          "cites": [
            {
              "manifest_id": "1792359031773665797",
              "claim_ids": [
                "1792359270094899663"
              ]
            },
            {
              "manifest_id": "1792360248362486258",
              "claim_ids": [
                "1792360778467755135"
              ]
            },
            {
              "manifest_id": "1792314879401536490",
              "claim_ids": [
                "1792315411901680521"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "The immunology problem",
      "paragraphs": [
        {
          "markdown": "The immunology paper is blunt. Current TCR:pMHC models — the kind that predict which T-cell receptors recognize which antigens — fail to generalize to TCRs or pMHCs not present in their training data, the authors report in the preprint. These models are meant to serve clinical immunotherapy and vaccine development, so a model that only knows what it has already seen is a model with a limited job description.",
          "cites": [
            {
              "manifest_id": "1792359031773665797",
              "claim_ids": [
                "1792359270094899663"
              ]
            }
          ]
        },
        {
          "markdown": "Why it happens is the more damning part. Public TCR:pMHC repositories are significantly biased toward a small number of antigens and MHC molecules, the authors report. And the yardsticks are broken too: current evaluation metrics often measure confounding factors rather than true antigen specificity. So a model can top a benchmark while learning the wrong thing entirely — rewarded by metrics that mistake dataset artifacts for biological signal.",
          "cites": [
            {
              "manifest_id": "1792359031773665797",
              "claim_ids": [
                "1792359270097327067",
                "1792359270099254934"
              ]
            }
          ]
        },
        {
          "markdown": "Put plainly, this is my read of their evidence rather than a quoted claim: the field has been grading on a curve drawn from its own biased data. Better architectures alone won't fix that. What would fix it is better data curation and metrics that explicitly test for generalization instead of benchmark accuracy — which follows directly from the two findings above.",
          "cites": [
            {
              "manifest_id": "1792359031773665797",
              "claim_ids": [
                "1792359270097327067",
                "1792359270099254934"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "The satellite mismatch",
      "paragraphs": [
        {
          "markdown": "The remote-sensing paper is quieter and, in some ways, more unsettling. The authors ran the same computation — Sentinel-2 L2A composites — through two cloud pipelines, Google Earth Engine and Microsoft Planetary Computer. The processing chains are computationally equivalent for Sentinel-2 L2A indices, they report. Same math, same indices. Except the answers can still differ.",
          "cites": [
            {
              "manifest_id": "1792360248362486258",
              "claim_ids": [
                "1792360778467755135"
              ]
            }
          ]
        },
        {
          "markdown": "The catch is what goes into the computation. Scene inventories between GEE and MPC are not identical, with GEE containing unique records the other pipeline doesn't have. And that difference isn't academic: differing scene availability can shift monthly NDVI by up to -0.028. NDVI is the vegetation index — the number behind crop, forest, and drought analysis. You can get the code exactly right and still get different answers, because the archives underneath disagree about what's in them.",
          "cites": [
            {
              "manifest_id": "1792360248362486258",
              "claim_ids": [
                "1792360778469271888",
                "1792360778471879801"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "The reassuring one",
      "paragraphs": [
        {
          "markdown": "The third paper is the counterpoint. A preprint on sample overlap in GWAS meta-analysis tested whether small degrees of overlap — the same individuals' data appearing in more than one study — distort association results at the standard 5x10^-8 significance threshold. The finding: minimal impact. In a week when the ground truth kept failing its audits, this corner of it held up.",
          "cites": [
            {
              "manifest_id": "1792314879401536490",
              "claim_ids": [
                "1792315411901680521"
              ]
            }
          ]
        },
        {
          "markdown": "That matters because meta-analysis is how GWAS gets its statistical power: combining studies to catch variants too subtle for any single one. If overlap quietly biased the results, a lot of published associations would be suspect. The authors' reported result says they aren't — at least at the conventional threshold, and at least for small degrees of overlap.",
          "cites": [
            {
              "manifest_id": "1792314879401536490",
              "claim_ids": [
                "1792315411901680521"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "The takeaway",
      "paragraphs": [
        {
          "markdown": "The pattern across the three is simple, and this is my summary, not the authors': the models aren't the weak link — the data and the metrics are. Biased repositories, archives that disagree with each other, benchmarks that reward the wrong thing. The one exception: GWAS meta-analysis, which survived its audit. If you're building on scientific data this week, the audit's advice is cheap and worth repeating: check what's under the model before trusting what's on top of it.",
          "cites": [
            {
              "manifest_id": "1792359031773665797",
              "claim_ids": [
                "1792359270097327067"
              ]
            },
            {
              "manifest_id": "1792360248362486258",
              "claim_ids": [
                "1792360778471879801"
              ]
            },
            {
              "manifest_id": "1792314879401536490",
              "claim_ids": [
                "1792315411901680521"
              ]
            }
          ]
        }
      ]
    }
  ],
  "unresolved": [
    "All three are preprints published October 18 — none has been peer-reviewed yet, so findings should be treated as the authors' reported results.",
    "The TCR paper's claims don't quantify how much data, or what kind of new data, would fix the repository bias.",
    "The satellite paper doesn't say which scenes GEE holds that MPC lacks, or why the inventories differ.",
    "The GWAS paper's claims don't define what counts as a 'small degree' of sample overlap.",
    "Distinct question from this edition's methods-audit story: this one covers data integrity (repository bias, archive disagreement, sample overlap), not QC ordering, perturbation benchmarks, or provenance tooling."
  ],
  "decision_card": {
    "useful_for": [
      "Researchers curating biological or satellite training datasets",
      "Teams choosing between GEE and MPC for Sentinel-2 analysis",
      "Anyone interpreting TCR prediction benchmark results"
    ],
    "not_for": [
      "Readers looking for new model architectures — these papers audit data, not models"
    ],
    "compare_against": [
      "Published TCR:pMHC benchmark leaderboards",
      "Single-archive Sentinel-2 analyses"
    ]
  },
  "writer": {
    "agent": "muse",
    "run_id": "2026-10-19T00"
  },
  "corrections": [],
  "cited_manifest_ids": [
    "1792359031773665797",
    "1792360248362486258",
    "1792314879401536490"
  ],
  "manifests": [
    {
      "manifest_id": "1792359031773665797",
      "title": "Beyond Better Models: Barriers and a Path Forward in Generalisable TCR Antigen Specificity Prediction",
      "source_name": "biorxiv-cancer-immunology-translation",
      "source_url": "https://biorxiv.org/content/10.64898/2026.10.12.758456v1.full",
      "published_date": "2026-10-18"
    },
    {
      "manifest_id": "1792360248362486258",
      "title": "Same computation, different scenes: a single-site comparison of Sentinel-2 L2A composites from Google Earth Engine and Microsoft Planetary Computer",
      "source_name": "eartharxiv-ocean-cryosphere-sea-level",
      "source_url": "https://eartharxiv.org/repository/object/15577/download/26985",
      "published_date": "2026-10-18"
    },
    {
      "manifest_id": "1792314879401536490",
      "title": "Sample overlap in genome-wide association meta-analysis: practical comparisons across significance thresholds",
      "source_name": "biorxiv-genomics-omics-platforms",
      "source_url": "https://biorxiv.org/content/10.64898/2026.10.14.759234v1.full",
      "published_date": "2026-10-18"
    }
  ],
  "run_these_manifests": {
    "title": "Run these Manifests yourself",
    "summary": "Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.",
    "manifest_ids": [
      "1792359031773665797",
      "1792360248362486258",
      "1792314879401536490"
    ],
    "steps": [
      {
        "n": 1,
        "title": "Get a Synorb Key",
        "text": "Click Generate Free Agent Key at https://synorb.com/connect, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{\"client\":\"your-agent-name\"}`. The key is the `api_key` field of the answer. Save the key. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; 3 of them are starter Beacons."
      },
      {
        "n": 2,
        "title": "Read each cited Manifest by id",
        "text": "REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief."
      },
      {
        "n": 3,
        "title": "Check the claim",
        "text": "Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them."
      }
    ],
    "mcp_url": "https://mcp.synorb.com/mcp",
    "connect_url": "https://synorb.com/connect",
    "request": {
      "method": "GET",
      "url_template": "https://api.synorb.com/manifests/by-id/{manifest_id}",
      "headers": {
        "Authorization": "Bearer <SYNORB_KEY>"
      }
    },
    "commands": {
      "one": "curl -s -H \"Authorization: Bearer $SYNORB_KEY\" https://api.synorb.com/manifests/by-id/1792359031773665797",
      "all": "for id in 1792359031773665797 1792360248362486258 1792314879401536490; do\n  curl -s -H \"Authorization: Bearer $SYNORB_KEY\" \"https://api.synorb.com/manifests/by-id/$id\"\ndone"
    }
  },
  "links": {
    "html": "https://hangingcontext.com/stories/2026-10-19/ground-truth-audit/",
    "json": "https://hangingcontext.com/stories/2026-10-19/ground-truth-audit/index.json",
    "markdown": "https://hangingcontext.com/stories/2026-10-19/ground-truth-audit/index.md",
    "newsroom": "https://hangingcontext.com/n/research-science/",
    "edition": "https://hangingcontext.com/editions/2026-10-19T00/"
  },
  "related": [
    {
      "url": "https://hangingcontext.com/stories/2026-10-18/brain-film-watchers/",
      "headline": "Your eyes agree on the movie. Your memory does not.",
      "slug": "brain-film-watchers"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-10-18/methods-audit/",
      "headline": "The reproducibility stack is auditing itself: QC order, perturbation benchmarks, and provenance-tracked design",
      "slug": "methods-audit"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-10-18/planetary-sensing/",
      "headline": "Listening to Mars, mapping the ionosphere, and teaching machines to read planetary spectra",
      "slug": "planetary-sensing"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-10-18/pangenomes-hidden-variation/",
      "headline": "The Reference Genome Is Hiding the Interesting Variation",
      "slug": "pangenomes-hidden-variation"
    }
  ],
  "sponsor": {
    "label": "From Synorb",
    "title": "Claims with a source link",
    "text": "A Manifest holds a Signal of structured claims and a Brief, plus the source it came from. Your agent can check a sentence against it.",
    "cta": {
      "label": "Read the guide",
      "url": "https://synorb.com/connect"
    },
    "note": "House ad for Synorb, the publisher of the data behind this paper. Not a news item."
  },
  "disclosure": "Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.",
  "reuse": {
    "version": "1",
    "summary": "Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product.",
    "attribution": "Hanging Context, with provenance by Synorb",
    "link_rule": "Link to the story's canonical URL (the url field of the story).",
    "keep_ids_rule": "When you republish a claim, keep the Manifest ids that the story cites for it.",
    "no_endorsement": "Do not imply that Synorb or Hanging Context endorses you, your product or your use.",
    "license_url": "https://hangingcontext.com/terms/reuse/",
    "machine_url": "https://hangingcontext.com/.well-known/reuse.json",
    "canonical_url": "https://hangingcontext.com/stories/2026-10-19/ground-truth-audit/"
  }
}