{
  "schema": "hangingcontext.desk/1",
  "type": "story",
  "id": "1862",
  "slug": "ai-researching-itself",
  "edition": "2026-11-21T00",
  "url": "https://hangingcontext.com/stories/2026-11-21/ai-researching-itself/",
  "newsroom": "artificial-intelligence",
  "beat": "research",
  "newsroom_title": "Artificial Intelligence",
  "angle": "Models now match experts on codable research tasks, but the benchmarks built to measure them are losing their grip.",
  "headline": "AI is starting to do the research, and the tests can barely keep up",
  "dek": "METR's latest update finds agent time horizons doubling every 3.6 months, while new benchmarks for AI scientists show frontier models scoring 45 out of 100 — and struggling to run their own training.",
  "published_at": "2026-11-21T00:43:51Z",
  "labels": {
    "trust": "confirmed",
    "trust_text": "Confirmed · 4 Manifests",
    "newsroom": "Artificial Intelligence"
  },
  "source_count": 4,
  "publishers": [
    "arxiv-ai-security-privacy-safety",
    "arxiv-frontier-methods-select",
    "huggingface-daily-papers",
    "metr-research"
  ],
  "sources": [
    "arxiv-ai-security-privacy-safety",
    "arxiv-frontier-methods-select",
    "huggingface-daily-papers",
    "metr-research"
  ],
  "distinct_source_count": 4,
  "key_facts": [
    {
      "text": "METR's Q4 2026 update finds frontier model time horizons doubling roughly every 3.6 months, with Opus 5.5 solving 77% of medium to large programs in the MirrorCode suite and GPT-6 Astra scoring 98% on FrontierMath Tier 4.",
      "cites": [
        {
          "claim_ids": [
            "1795206518450578610",
            "1795206518451578611",
            "1795206518455578616",
            "1795206518456578617"
          ],
          "manifest_id": "1795206231914574775"
        }
      ],
      "self_reported": false
    },
    {
      "text": "In a preprint, TwinLab-Bench tests AI scientists on 52 tasks drawn from real laboratory records, and the strongest model evaluated, Claude Fable 5.2, averaged a score of 45.0.",
      "cites": [
        {
          "claim_ids": [
            "1795180810249395844",
            "1795180810253395853",
            "1795180810255395855"
          ],
          "manifest_id": "1795180492910393726"
        }
      ],
      "self_reported": true
    },
    {
      "text": "In a preprint, Agent^2 RL-Bench shows agents improving ALFWorld performance from 4.85 to 93.28 using supervised warm-up and online rollouts, yet stable agent-driven RL post-training remains rare under fixed budgets.",
      "cites": [
        {
          "claim_ids": [
            "1795153412244243779",
            "1795153412245243780"
          ],
          "manifest_id": "1795152988385238050"
        }
      ],
      "self_reported": true
    },
    {
      "text": "In a preprint, Paper2Lean formalised 20 mathematical optimisation papers with 100% compilation success, producing 110,910 lines of verified Lean code.",
      "cites": [
        {
          "claim_ids": [
            "1795162818435318135",
            "1795162818436318137"
          ],
          "manifest_id": "1795162454624311709"
        }
      ],
      "self_reported": true
    }
  ],
  "images": [],
  "sections": [
    {
      "heading": "The capability curve",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1795206518450578610",
                "1795206518451578611",
                "1795206518455578616",
                "1795206518456578617"
              ],
              "manifest_id": "1795206231914574775"
            }
          ],
          "markdown": "METR's Q4 2026 update puts numbers on a trend that has been hard to argue with. Frontier models now match or outperform skilled humans on coding and optimisation tasks, and the time horizons agents can handle are doubling roughly every 3.6 months. Opus 5.5 solves 77 percent of medium to large programs in the MirrorCode suite, and GPT-6 Astra scored 98 percent on FrontierMath Tier 4."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1795206518452578612",
                "1795206518453578613",
                "1795206518454578615"
              ],
              "manifest_id": "1795206231914574775"
            }
          ],
          "markdown": "The same update draws a line the models have not crossed. They do worse than human experts on judgement tasks, the kind where choices have no clear empirical feedback, and they do worse on tasks with higher lifecycle cost, even when task length is controlled for. METR says current public evidence is insufficient to rule out drastic AI research acceleration in the next couple of years, which is a careful way of saying the acceleration is plausible but unproven."
        }
      ]
    },
    {
      "heading": "The tests are struggling too",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1795180810246395840",
                "1795180810249395844",
                "1795180810252395850",
                "1795180810253395853",
                "1795180810255395855"
              ],
              "manifest_id": "1795180492910393726"
            }
          ],
          "markdown": "A preprint introducing TwinLab-Bench tries to measure the thing METR is warning about: AI agents acting as scientists. The benchmark covers 52 tasks from physical sciences, life sciences and engineering, and tests three abilities, workflow execution, measurement selection and experimental design. Each task uses a TwinLab agent built from real laboratory records to execute the experiments the tested agent designs. The strongest model evaluated, Claude Fable 5.2, averaged a score of 45.0."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1795153412241243778",
                "1795153412244243779",
                "1795153412245243780"
              ],
              "manifest_id": "1795152988385238050"
            },
            {
              "claim_ids": [
                "1795162818435318135",
                "1795162818436318136",
                "1795162818436318137"
              ],
              "manifest_id": "1795162454624311709"
            }
          ],
          "markdown": "Two more preprints fill in the picture. Agent^2 RL-Bench asks whether agents can run their own reinforcement-learning post-training end to end, and finds they can lift ALFWorld performance from 4.85 to 93.28 with supervised warm-up and online rollouts, but stable agent-driven training under fixed budgets remains rare. Paper2Lean, an agentic system that turns maths papers into formal Lean proofs, reached 100 percent compilation success on 20 optimisation papers and produced 110,910 lines of verified Lean code, yet semantic fidelity remains a central challenge even when the proofs verify."
        }
      ]
    },
    {
      "heading": "The judge problem",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1795206518454578614",
                "1795206518458578619"
              ],
              "manifest_id": "1795206231914574775"
            }
          ],
          "markdown": "There is a second-order problem hiding in the measurement. METR finds that AI judges systematically overestimate the quality of tasks made by models stronger than themselves, and that agents now build borderline-useful evaluation tasks, but not yet ones METR considers especially informative. If the judge cannot tell a good test from a bad one, the loop that is supposed to improve the model starts grading its own homework."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1795206518452578612",
                "1795206518453578613"
              ],
              "manifest_id": "1795206231914574775"
            },
            {
              "claim_ids": [
                "1795180810256395859"
              ],
              "manifest_id": "1795180492910393726"
            }
          ],
          "markdown": "TwinLab-Bench tries to defend against this by grounding its environments in real laboratory records and using closed-book probes to control for answers recalled from memory. The design choice matters: as training environments increasingly resemble testing benchmarks, METR warns that current evaluation methods may be losing their predictive power. A benchmark that a model can pass by memorising the lab is not measuring research ability."
        }
      ]
    },
    {
      "heading": "What it means",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1795206518451578611",
                "1795206518452578612"
              ],
              "manifest_id": "1795206231914574775"
            },
            {
              "claim_ids": [
                "1795180810255395855"
              ],
              "manifest_id": "1795180492910393726"
            }
          ],
          "markdown": "For anyone running a lab, the practical split is now clear. Hand agents the verifiable work, coding, optimisation, formalisation, where they already match or beat skilled humans. Keep humans on the judgement calls, the experimental design and the interpretation, where the best model in the hardest test scores 45 out of 100. That is not a temporary gap. It is the shape of the capability curve as it stands."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1795206518454578615",
                "1795206518457578618"
              ],
              "manifest_id": "1795206231914574775"
            }
          ],
          "markdown": "What to watch is whether the next update closes the gap on high lifecycle cost tasks, the expensive, slow-feedback work that METR uses as a proxy for evaluation difficulty. METR says it remains uncertain about the relative growth rates there. Until those tasks start moving, talk of runaway research acceleration is a forecast, not a fact."
        }
      ]
    }
  ],
  "unresolved": [
    "Whether time-horizon growth continues at the current rate on high lifecycle cost tasks",
    "Whether TwinLab-Bench scores improve once models are trained specifically for lab work",
    "Whether agent-driven RL post-training becomes stable without supervised warm-up pipelines"
  ],
  "decision_card": {
    "not_for": [
      "Anyone who needs a model to make judgement calls without human review",
      "Teams that take a benchmark score as proof a model can do real research"
    ],
    "useful_for": [
      "Lab leaders planning how much R&D to hand to agents",
      "Executives deciding where human judgement still has to sit in the loop"
    ],
    "compare_against": [
      "Hiring human researchers for well-defined coding and optimisation work",
      "Existing benchmarks that score models on fixed, verifiable tasks"
    ]
  },
  "writer": {
    "agent": "muse",
    "run_id": "2026-11-21T00-batch-ai-1"
  },
  "corrections": [],
  "cited_manifest_ids": [
    "1795206231914574775",
    "1795180492910393726",
    "1795152988385238050",
    "1795162454624311709"
  ],
  "manifests": [
    {
      "title": "AI R&D-relevant model capabilities are improving rapidly and are getting harder to measure",
      "source_url": "https://metr.org/blog/2026-11-20-q4-2026-capability-update",
      "manifest_id": "1795206231914574775",
      "source_name": "metr-research",
      "published_date": "2026-11-20"
    },
    {
      "title": "TwinLab-Bench: Evaluating AI Scientists in Agentic Twins of Real Labs",
      "source_url": "https://huggingface.co/papers/2611.26248",
      "manifest_id": "1795180492910393726",
      "source_name": "huggingface-daily-papers",
      "published_date": "2026-11-20"
    },
    {
      "title": "Agent^2 RL-Bench: Can LLM Agents Engineer Agentic RL Post-Training?",
      "source_url": "https://arxiv.org/pdf/2604.10547v3",
      "manifest_id": "1795152988385238050",
      "source_name": "arxiv-ai-security-privacy-safety",
      "published_date": "2026-11-20"
    },
    {
      "title": "Paper2Lean: Agentic Autoformalization of Mathematical Research Papers at Scale",
      "source_url": "https://arxiv.org/pdf/2611.26272v1",
      "manifest_id": "1795162454624311709",
      "source_name": "arxiv-frontier-methods-select",
      "published_date": "2026-11-20"
    }
  ],
  "run_these_manifests": {
    "title": "Run these Manifests yourself",
    "summary": "Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.",
    "manifest_ids": [
      "1795206231914574775",
      "1795180492910393726",
      "1795152988385238050",
      "1795162454624311709"
    ],
    "steps": [
      {
        "n": 1,
        "title": "Get a Synorb Key",
        "text": "Click Generate Free Agent Key at https://synorb.com/connect/start, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{\"client\":\"your-agent-name\"}`. The key is the `api_key` field of the answer. Save the key. Existing 72 Hour Agent Demo keys keep their issued limits. New keys require email OTP verification. An existing demo key lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons."
      },
      {
        "n": 2,
        "title": "Read each cited Manifest by id",
        "text": "REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief."
      },
      {
        "n": 3,
        "title": "Check the claim",
        "text": "Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them."
      }
    ],
    "mcp_url": "https://mcp.synorb.com/mcp",
    "connect_url": "https://synorb.com/connect/start",
    "request": {
      "method": "GET",
      "url_template": "https://api.synorb.com/manifests/by-id/{manifest_id}",
      "headers": {
        "Authorization": "Bearer <SYNORB_KEY>"
      }
    },
    "commands": {
      "one": "curl -s -H \"Authorization: Bearer $SYNORB_KEY\" https://api.synorb.com/manifests/by-id/1795206231914574775",
      "all": "for id in 1795206231914574775 1795180492910393726 1795152988385238050 1795162454624311709; do\n  curl -s -H \"Authorization: Bearer $SYNORB_KEY\" \"https://api.synorb.com/manifests/by-id/$id\"\ndone"
    }
  },
  "links": {
    "html": "https://hangingcontext.com/stories/2026-11-21/ai-researching-itself/",
    "json": "https://hangingcontext.com/stories/2026-11-21/ai-researching-itself/index.json",
    "markdown": "https://hangingcontext.com/stories/2026-11-21/ai-researching-itself/index.md",
    "newsroom": "https://hangingcontext.com/n/artificial-intelligence/",
    "edition": "https://hangingcontext.com/editions/2026-11-21T00/"
  },
  "related": [
    {
      "url": "https://hangingcontext.com/stories/2026-11-21/models-rewrite-own-weights/",
      "headline": "The model is learning to edit itself, and the edits stick",
      "slug": "models-rewrite-own-weights"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-11-21/the-checkers-cannot-keep-up/",
      "headline": "We cannot tell fake from real, and our checkers cannot either",
      "slug": "the-checkers-cannot-keep-up"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-11-22/enterprises-want-ai-on-their-terms/",
      "headline": "Enterprises are building private, curated AI — not buying another chatbot",
      "slug": "enterprises-want-ai-on-their-terms"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-11-21/legal-ai-cant-prove-it/",
      "headline": "Legal adopted AI fastest. Now it has to prove the money was well spent",
      "slug": "legal-ai-cant-prove-it"
    }
  ],
  "sponsor": {
    "label": "From Synorb",
    "title": "Run the cited Manifests yourself",
    "text": "Every claim in this paper points to a Manifest id. Get a Synorb Key and read the Manifests behind it. Existing 72 Hour Agent Demo keys keep their issued limits. New keys require email OTP verification. An existing demo key lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons.",
    "cta": {
      "label": "Get a key",
      "url": "https://synorb.com/connect/start"
    },
    "note": "House ad for Synorb, the publisher of the data behind this paper. Not a news item."
  },
  "disclosure": "Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.",
  "reuse": {
    "version": "1",
    "summary": "Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product.",
    "attribution": "Hanging Context, with provenance by Synorb",
    "link_rule": "Link to the story's canonical URL (the url field of the story).",
    "keep_ids_rule": "When you republish a claim, keep the Manifest ids that the story cites for it.",
    "no_endorsement": "Do not imply that Synorb or Hanging Context endorses you, your product or your use.",
    "license_url": "https://hangingcontext.com/terms/reuse/",
    "machine_url": "https://hangingcontext.com/.well-known/reuse.json",
    "canonical_url": "https://hangingcontext.com/stories/2026-11-21/ai-researching-itself/"
  }
}
