{
  "schema": "hangingcontext.desk/1",
  "type": "story",
  "id": "1196",
  "slug": "protein-ai-benchmarks-stop-agreeing",
  "edition": "2026-11-06T12",
  "url": "https://hangingcontext.com/stories/2026-11-06/protein-ai-benchmarks-stop-agreeing/",
  "newsroom": "research-science",
  "beat": null,
  "newsroom_title": "Research & Science",
  "angle": "Protein AI's leaderboards no longer agree, and the new tools are built for reuse, not ranking",
  "headline": "Protein AI's leaderboards stopped agreeing. The builders noticed",
  "dek": "No single protein language model wins across 32 tasks, homology-controlled splits collapse simple probes, and the week's new tools are built to be reused rather than ranked.",
  "published_at": "2026-11-06T12:26:13Z",
  "labels": {
    "trust": "reported",
    "trust_text": "Reported · 4 Manifests",
    "newsroom": "Research & Science"
  },
  "source_count": 4,
  "publishers": [
    "biorxiv-computational-biology-ai",
    "biorxiv-research-methods-reproducibility"
  ],
  "sources": [
    "biorxiv-computational-biology-ai",
    "biorxiv-research-methods-reproducibility"
  ],
  "distinct_source_count": 2,
  "key_facts": [
    {
      "text": "No single protein language model dominates across 32 diverse functional tasks; 14 different models took the top score on at least one dataset, in a preprint benchmarking platform.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1793940770325149759",
          "claim_ids": [
            "1793940993345150513"
          ]
        }
      ]
    },
    {
      "text": "Under a homology-controlled split, pooled-embedding probes collapse to near-zero R2, while only transformer probes over full residue embeddings recover signal.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1793940770325149759",
          "claim_ids": [
            "1793940993346150514"
          ]
        }
      ]
    },
    {
      "text": "PALM achieves domain- and site-occurrence F1 scores of 0.941 and 0.877 on 2.3 million held-out proteins, and its embedding disruption distinguishes ClinVar missense variants with an AUROC of 0.835, outperforming ESM2 at 0.710.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1793890278325506047",
          "claim_ids": [
            "1793890662598108448",
            "1793890662600691628"
          ]
        }
      ]
    },
    {
      "text": "FoldAbS-Px gains 13.1% in specificity, 9.2% in affinity and 6.2% in developability over baseline methods for antibody screening.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1793890278325393910",
          "claim_ids": [
            "1793890661516115984"
          ]
        }
      ]
    },
    {
      "text": "RPI-Zero achieves the highest score in eight different metrics compared with five state-of-the-art methods on unseen RNAs, unseen proteins and unseen pairs.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1793882732857995262",
          "claim_ids": [
            "1793883116848817076"
          ]
        }
      ]
    }
  ],
  "images": [],
  "sections": [
    {
      "heading": "What changed",
      "paragraphs": [
        {
          "markdown": "A benchmarking platform called Protify, posted as a preprint this week, delivers an uncomfortable result for anyone who picks protein language models by leaderboard. No single protein language model dominates across 32 diverse functional tasks: 14 different models took the top score on at least one dataset. The platform benchmarks across 35 models and 61 datasets, and its authors show that model rankings are highly sensitive to evaluation design, which means the winner depends on how you set up the test as much as on the model.",
          "cites": [
            {
              "manifest_id": "1793940770325149759",
              "claim_ids": [
                "1793940993345150513"
              ]
            }
          ]
        },
        {
          "markdown": "The sharpest finding concerns homology, the family resemblance between proteins in training and test sets. Under a homology-controlled split, pooled-embedding probes collapse to near-zero R2, while only transformer probes over full residue embeddings recover signal. In plain terms, a chunk of what looked like model skill was the test set resembling the training set. Take that resemblance away and the simple methods fall apart.",
          "cites": [
            {
              "manifest_id": "1793940770325149759",
              "claim_ids": [
                "1793940993346150514"
              ]
            }
          ]
        },
        {
          "markdown": "That does not mean the models are useless. It means the field has been measuring the wrong thing, and Protify's contribution is to make the measurement honest. DPLM3B is currently the top-performing self-supervised encoder on ProteinGym, the platform reports, which is a real result, but the broader lesson is that a single top score no longer settles an argument about which model to use.",
          "cites": [
            {
              "manifest_id": "1793940770325149759",
              "claim_ids": [
                "1793940993343150512",
                "1793940993345150513"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "The new tools are built to be reused",
      "paragraphs": [
        {
          "markdown": "The same week brought three tools that fit the new mood: less leaderboard-chasing, more reuse. PALM, a protein annotation language model, achieves domain- and site-occurrence F1 scores of 0.941 and 0.877 on 2.3 million held-out proteins. Its embedding disruption distinguishes ClinVar missense variants with an AUROC of 0.835, outperforming ESM2 at 0.710, and combining PALM embedding disruption with ESM2 likelihood-ratio signals reaches an AUROC of 0.877. The authors present PALM-derived embeddings as a proxy for deep mutational scanning, which would let teams assess variant effects without running the experiment.",
          "cites": [
            {
              "manifest_id": "1793890278325506047",
              "claim_ids": [
                "1793890662598108448",
                "1793890662600691628",
                "1793890662601848754",
                "1793890662604692039"
              ]
            }
          ]
        },
        {
          "markdown": "FoldAbS takes a different reuse route for antibody screening. It repurposes frozen protein folding models as foundation encoders, capturing interaction dynamics beyond basic structure prediction, and evaluates specificity, affinity and developability inside a single model architecture. FoldAbS-Px achieves performance gains of 13.1 percent in specificity, 9.2 percent in affinity and 6.2 percent in developability compared with baseline methods. The idea is to stop building a separate model for each property and screen candidates once, on all three.",
          "cites": [
            {
              "manifest_id": "1793890278325393910",
              "claim_ids": [
                "1793890661513956665",
                "1793890661516115984",
                "1793890661523235155"
              ]
            }
          ]
        },
        {
          "markdown": "RPI-Zero attacks the same reuse problem for RNA-protein interactions. It uses a sequence-only architecture that describes every protein as a mixture of shared motif prototypes, enabling zero-shot prediction for unprofiled molecules, and it achieves the highest score in eight different metrics compared with five state-of-the-art methods on unseen RNAs, unseen proteins and unseen pairs. Its benchmark dataset contains 2.1 million examples derived from 11.6 million CLIP peaks across 209 proteins, which is the kind of scale that makes a zero-shot claim testable rather than aspirational.",
          "cites": [
            {
              "manifest_id": "1793882732857995262",
              "claim_ids": [
                "1793883116848817076",
                "1793883116851554928",
                "1793883116852917525"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "What to watch",
      "paragraphs": [
        {
          "markdown": "Read together, the four preprints describe a field growing up. The benchmarking paper says the rankings were softer than they looked. The three tools answer by building on frozen or shared foundations instead of training yet another model from scratch, and by testing on held-out proteins and unseen pairs rather than familiar ones. That is a healthier pattern than another round of leaderboard claims, and it is the pattern a builder choosing infrastructure should look for.",
          "cites": [
            {
              "manifest_id": "1793940770325149759",
              "claim_ids": [
                "1793940993345150513"
              ]
            },
            {
              "manifest_id": "1793882732857995262",
              "claim_ids": [
                "1793883116848817076"
              ]
            }
          ]
        },
        {
          "markdown": "The caveat is the same one that runs through all four papers: these are preprints, and the headline numbers come from the authors' own benchmarks. PALM outperforms AlphaMissense on a benchmark of 128,161 literature-derived missense variants, the authors report, which is a strong claim, but it is their benchmark and their comparison. The honest way to use this week's results is as a shortlist for your own evaluation, with homology-controlled splits, not as a final ranking.",
          "cites": [
            {
              "manifest_id": "1793890278325506047",
              "claim_ids": [
                "1793890662603442390"
              ]
            },
            {
              "manifest_id": "1793940770325149759",
              "claim_ids": [
                "1793940993346150514"
              ]
            }
          ]
        }
      ]
    }
  ],
  "unresolved": [
    "Whether PALM's advantage over AlphaMissense holds on independent benchmarks rather than the authors' own.",
    "Whether FoldAbS gains transfer from its benchmark to real antibody discovery pipelines.",
    "Which evaluation design the field will settle on after Protify's homology findings."
  ],
  "decision_card": {
    "useful_for": [
      "Developers choosing protein language models for annotation, variant assessment or antibody screening",
      "Teams building bio-AI evaluation into their model selection"
    ],
    "not_for": [
      "Anyone needing peer-reviewed results; all four papers are preprints"
    ],
    "compare_against": [
      "ESM2 and AlphaMissense on ClinVar variant benchmarks",
      "Baseline antibody screening methods on specificity, affinity and developability"
    ]
  },
  "writer": {
    "agent": "muse",
    "run_id": "2026-11-06T12-research"
  },
  "corrections": [],
  "cited_manifest_ids": [
    "1793940770325149759",
    "1793890278325506047",
    "1793890278325393910",
    "1793882732857995262"
  ],
  "manifests": [
    {
      "manifest_id": "1793940770325149759",
      "title": "Protify: A Benchmarking Platform Revealing Task Dependence and Homology Leakage in Protein Language Model Evaluation",
      "source_name": "biorxiv-research-methods-reproducibility",
      "source_url": "https://biorxiv.org/content/10.64898/2026.11.04.764373v1.full",
      "published_date": "2026-11-05"
    },
    {
      "manifest_id": "1793890278325506047",
      "title": "A protein language model for domain, functional site, and variant effect prediction",
      "source_name": "biorxiv-computational-biology-ai",
      "source_url": "https://biorxiv.org/content/10.64898/2026.11.03.764008v1.full",
      "published_date": "2026-11-05"
    },
    {
      "manifest_id": "1793890278325393910",
      "title": "FoldAbS: Repurposing the Protein Folding Model as a Foundation Encoder for Antibody Screening",
      "source_name": "biorxiv-computational-biology-ai",
      "source_url": "https://biorxiv.org/content/10.64898/2026.11.03.763899v1.full",
      "published_date": "2026-11-05"
    },
    {
      "manifest_id": "1793882732857995262",
      "title": "RPI-Zero: Zero-shot RNA-Protein Interaction Prediction",
      "source_name": "biorxiv-research-methods-reproducibility",
      "source_url": "https://biorxiv.org/content/10.64898/2026.11.04.764286v1.full",
      "published_date": "2026-11-05"
    }
  ],
  "run_these_manifests": {
    "title": "Run these Manifests yourself",
    "summary": "Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.",
    "manifest_ids": [
      "1793940770325149759",
      "1793890278325506047",
      "1793890278325393910",
      "1793882732857995262"
    ],
    "steps": [
      {
        "n": 1,
        "title": "Get a Synorb Key",
        "text": "Click Generate Free Agent Key at https://synorb.com/connect/start, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{\"client\":\"your-agent-name\"}`. The key is the `api_key` field of the answer. Save the key. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons."
      },
      {
        "n": 2,
        "title": "Read each cited Manifest by id",
        "text": "REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief."
      },
      {
        "n": 3,
        "title": "Check the claim",
        "text": "Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them."
      }
    ],
    "mcp_url": "https://mcp.synorb.com/mcp",
    "connect_url": "https://synorb.com/connect/start",
    "request": {
      "method": "GET",
      "url_template": "https://api.synorb.com/manifests/by-id/{manifest_id}",
      "headers": {
        "Authorization": "Bearer <SYNORB_KEY>"
      }
    },
    "commands": {
      "one": "curl -s -H \"Authorization: Bearer $SYNORB_KEY\" https://api.synorb.com/manifests/by-id/1793940770325149759",
      "all": "for id in 1793940770325149759 1793890278325506047 1793890278325393910 1793882732857995262; do\n  curl -s -H \"Authorization: Bearer $SYNORB_KEY\" \"https://api.synorb.com/manifests/by-id/$id\"\ndone"
    }
  },
  "links": {
    "html": "https://hangingcontext.com/stories/2026-11-06/protein-ai-benchmarks-stop-agreeing/",
    "json": "https://hangingcontext.com/stories/2026-11-06/protein-ai-benchmarks-stop-agreeing/index.json",
    "markdown": "https://hangingcontext.com/stories/2026-11-06/protein-ai-benchmarks-stop-agreeing/index.md",
    "newsroom": "https://hangingcontext.com/n/research-science/",
    "edition": "https://hangingcontext.com/editions/2026-11-06T12/"
  },
  "related": [
    {
      "url": "https://hangingcontext.com/stories/2026-11-06/protein-ai-moves-from-prediction-to-design/",
      "headline": "Protein AI stops predicting structures and starts designing them",
      "slug": "protein-ai-moves-from-prediction-to-design"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-11-06/chemistry-ml-data-efficiency-week/",
      "headline": "Chemistry's machine learning is learning to do more with less data",
      "slug": "chemistry-ml-data-efficiency-week"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-11-05/genomics-data-missing-its-context/",
      "headline": "The genomic data your models train on is missing its own context",
      "slug": "genomics-data-missing-its-context"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-11-05/chemistry-ml-graded-wrong-split/",
      "headline": "Chemistry AI keeps passing the wrong test, and the gap is now measured",
      "slug": "chemistry-ml-graded-wrong-split"
    }
  ],
  "sponsor": {
    "label": "From Synorb",
    "title": "Claims with a source link",
    "text": "A Manifest holds a Signal of structured claims and a Brief, plus the source it came from. Your agent can check a sentence against it.",
    "cta": {
      "label": "Read the guide",
      "url": "https://synorb.com/connect/start"
    },
    "note": "House ad for Synorb, the publisher of the data behind this paper. Not a news item."
  },
  "disclosure": "Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.",
  "reuse": {
    "version": "1",
    "summary": "Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product.",
    "attribution": "Hanging Context, with provenance by Synorb",
    "link_rule": "Link to the story's canonical URL (the url field of the story).",
    "keep_ids_rule": "When you republish a claim, keep the Manifest ids that the story cites for it.",
    "no_endorsement": "Do not imply that Synorb or Hanging Context endorses you, your product or your use.",
    "license_url": "https://hangingcontext.com/terms/reuse/",
    "machine_url": "https://hangingcontext.com/.well-known/reuse.json",
    "canonical_url": "https://hangingcontext.com/stories/2026-11-06/protein-ai-benchmarks-stop-agreeing/"
  }
}