{
  "schema": "hangingcontext.desk/1",
  "type": "story",
  "id": "518",
  "slug": "llm-judges-unstable",
  "edition": "2026-10-24T00",
  "url": "https://hangingcontext.com/stories/2026-10-24/llm-judges-unstable/",
  "newsroom": "artificial-intelligence",
  "beat": "research",
  "newsroom_title": "Artificial Intelligence",
  "angle": "LLM-as-a-judge is unstable across prompting strategies and languages; the field answers with mechanical verification.",
  "headline": "LLM judges change verdicts with the prompting strategy — so the field moves to mechanical verification",
  "dek": "Prompting strategies don't transfer across judge models or languages. The responses: check work mechanically, tune evaluation prompts instead of writing them, and price classification mistakes.",
  "published_at": "2026-10-24T01:03:41Z",
  "labels": {
    "trust": "reported",
    "trust_text": "Reported · 4 sources",
    "newsroom": "Artificial Intelligence"
  },
  "source_count": 4,
  "publishers": [
    "arxiv-ai-security-privacy-safety",
    "arxiv-llm-evals-benchmarks"
  ],
  "key_facts": [
    {
      "text": "In a preprint, the authors report that prompting strategies are not transferable across judge models and languages, and that the choice of judge model dominates the choice of prompting strategy in evaluation outcomes.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1792735075842543379",
          "claim_ids": [
            "1792735445215910990",
            "1792735445222135090"
          ]
        }
      ]
    },
    {
      "text": "In the same preprint, the authors report that evaluation stability in LLM-as-a-judge systems degrades significantly in low-resource languages.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1792735075842543379",
          "claim_ids": [
            "1792735445227313296"
          ]
        }
      ]
    },
    {
      "text": "In a preprint, Anna Mei and Michael R. Douglas report that combining LLM semantic review with mechanical numerical and vacuity checks reduces the accepted-error rate in Lean formalizations versus LLM-only verification.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1792737665293761959",
          "claim_ids": [
            "1792737994170767750"
          ]
        }
      ]
    },
    {
      "text": "In a preprint, the authors report that soft-prompt tuning saturates format-following within 150 steps (about 1,200 samples) while preserving knowledge accuracy, and matches strong evaluation-harness baselines without manual prompt or parser design.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1792737665293123822",
          "claim_ids": [
            "1792737988539154923",
            "1792737988551523926"
          ]
        }
      ]
    },
    {
      "text": "In a preprint on IT ticket classification, the author reports that TF-IDF with logistic regression at a 0.60 threshold reduced test cost by 38.1% and wrong routes from 15.3% to 3.5%.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1792737665293214113",
          "claim_ids": [
            "1792737989527275848"
          ]
        }
      ]
    }
  ],
  "images": [],
  "sections": [
    {
      "heading": "The judge is the variable",
      "paragraphs": [
        {
          "markdown": "If you evaluate models with models, you have probably done prompt engineering on the judge. A multilingual study by Asal Mehrabi, Mohammad Aliannejadi and Clemencia Siro, posted as a preprint this week, suggests that work may not travel: prompting strategies like chain-of-thought and role conditioning do not transfer reliably across judge models or languages, and the choice of judge model is the dominant factor in evaluation outcomes, overriding the choice of prompting strategy. You are not tuning the prompt so much as tuning to the judge.",
          "cites": [
            {
              "manifest_id": "1792735075842543379",
              "claim_ids": [
                "1792735445215910990",
                "1792735445222135090"
              ]
            }
          ]
        },
        {
          "markdown": "The instability gets worse where it matters most for global deployments: evaluation stability degrades significantly in low-resource languages. That means aggregate multilingual scores can hide per-language failures — a single average number may be averaging away exactly the cases you care about.",
          "cites": [
            {
              "manifest_id": "1792735075842543379",
              "claim_ids": [
                "1792735445227313296"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "Replace the verdict with a machine check",
      "paragraphs": [
        {
          "markdown": "One response is to stop asking the model to judge at all and check the work mechanically. Anna Mei and Michael R. Douglas take this route for autoformalization — translating mathematics into Lean proofs — where they combine LLM-based semantic review with mechanical numerical and vacuity checks. The hybrid approach reduces the accepted-error rate in Lean formalizations compared to LLM-only verification. Alongside it, the authors built the Formalized Library of Mathematical Functions (FLMF), a partial formalization of NIST DLMF content, as a benchmark for evaluating autoformalization methods.",
          "cites": [
            {
              "manifest_id": "1792737665293761959",
              "claim_ids": [
                "1792737994170767750",
                "1792737994172456659"
              ]
            }
          ]
        },
        {
          "markdown": "The lesson generalizes past mathematics: anywhere an LLM's verdict can be checked by computation rather than by another LLM, the mechanical check is the cheaper source of truth. High-assurance agentic pipelines should be treating LLM judges as the first pass and the mechanical check as the verdict.",
          "cites": [
            {
              "manifest_id": "1792737665293761959",
              "claim_ids": [
                "1792737994170767750"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "Tune the evaluation, don't write it",
      "paragraphs": [
        {
          "markdown": "A second response keeps the LLM in the loop but removes the prompt-engineering tax on evaluation itself. In a preprint, Selen Erkan and colleagues propose soft-prompt tuning: instead of hand-crafting prompts and parsers for each benchmark, they adapt models to benchmark formats through tuned soft prompts. Format-following saturates within 150 steps — about 1,200 samples — while knowledge accuracy is preserved, and the method matches strong evaluation-harness baselines without any manual benchmark-specific prompt or parser design. It also improves the evaluation signal on long-context datasets and instruction-tuned variants with different output styles.",
          "cites": [
            {
              "manifest_id": "1792737665293123822",
              "claim_ids": [
                "1792737988539154923",
                "1792737988551523926",
                "1792737988553305852"
              ]
            }
          ]
        },
        {
          "markdown": "The throughline is familiar: evaluation pipelines keep failing on the same brittleness — the prompt is a variable nobody controls. Tuning the evaluation prompt rather than hand-writing it is the obvious next step, and it is a small one compared to retraining a judge.",
          "cites": [
            {
              "manifest_id": "1792737665293123822",
              "claim_ids": [
                "1792737988551523926"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "Price the mistakes instead",
      "paragraphs": [
        {
          "markdown": "A third paper asks the cost question directly. In a preprint on LLM-based IT ticket classification, Sachin Gupta adds abstention to the picture: the model can hand a ticket to a human when it is unsure, and the routing policy is calibrated on actual costs. The measured result is striking — TF-IDF with logistic regression at a 0.60 threshold reduced test cost by 38.1% and wrong routes from 15.3% to 3.5%, and calibration-selected policies beat always-route and always-defer strategies in 14 of 16 tested configurations. The paper also finds that isotonic recalibration lowers expected calibration error without reliably improving discrimination: the standard calibration metric is not the operational one.",
          "cites": [
            {
              "manifest_id": "1792737665293214113",
              "claim_ids": [
                "1792737989527275848",
                "1792737989534953160",
                "1792737989551783824"
              ]
            }
          ]
        },
        {
          "markdown": "The quiet punchline: on a cost-sensitive task, a linear model with a threshold outperformed the instinct to throw a bigger LLM at the problem. The arc of the week is the same across all four papers — the judge is unreliable, so verify mechanically, tune the evaluation prompts, or price the mistakes. The era of trusting a single LLM's verdict is ending; the era of instrumenting it is here.",
          "cites": [
            {
              "manifest_id": "1792735075842543379",
              "claim_ids": []
            },
            {
              "manifest_id": "1792737665293761959",
              "claim_ids": []
            },
            {
              "manifest_id": "1792737665293123822",
              "claim_ids": []
            },
            {
              "manifest_id": "1792737665293214113",
              "claim_ids": []
            }
          ]
        }
      ]
    }
  ],
  "unresolved": [
    "All four papers are preprints without independent replication.",
    "The multilingual judge study does not establish how much of the instability comes from the judges versus the datasets used.",
    "The IT-ticket findings are scoped to one dataset and prompt design; cost-optimal thresholds will differ elsewhere."
  ],
  "decision_card": {
    "useful_for": [
      "AI developers running LLM-as-a-judge evaluations",
      "teams building multilingual evaluation or high-assurance agentic pipelines"
    ],
    "not_for": [
      "readers looking for new model releases — this is evaluation research",
      "teams whose evals already run through calibrated, mechanically checked pipelines"
    ],
    "compare_against": [
      "LLM-only verification baselines",
      "manually prompt-engineered evaluation harnesses",
      "always-route and always-defer routing policies"
    ]
  },
  "writer": {
    "agent": "muse",
    "run_id": "2026-10-24T00"
  },
  "corrections": [],
  "cited_manifest_ids": [
    "1792735075842543379",
    "1792737665293761959",
    "1792737665293123822",
    "1792737665293214113"
  ],
  "manifests": [
    {
      "manifest_id": "1792735075842543379",
      "title": "Prompting Strategy Effects Depend on the Judge: A Multilingual Study of LLM-as-a-Judge Robustness",
      "source_name": "arxiv-ai-security-privacy-safety",
      "source_url": "https://arxiv.org/pdf/2610.31172v1",
      "published_date": "2026-10-23"
    },
    {
      "manifest_id": "1792737665293761959",
      "title": "Beyond LLM Judge: Mechanical Verification for Agentic Autoformalization",
      "source_name": "arxiv-llm-evals-benchmarks",
      "source_url": "https://arxiv.org/pdf/2610.20573v2",
      "published_date": "2026-10-23"
    },
    {
      "manifest_id": "1792737665293123822",
      "title": "Beyond Prompt and Parser Engineering: Soft-Prompt Tuning for Automated LLM Evaluation",
      "source_name": "arxiv-llm-evals-benchmarks",
      "source_url": "https://arxiv.org/pdf/2606.12117v2",
      "published_date": "2026-10-23"
    },
    {
      "manifest_id": "1792737665293214113",
      "title": "Cost-Aware Abstention for LLM-Based IT Ticket Classification",
      "source_name": "arxiv-llm-evals-benchmarks",
      "source_url": "https://arxiv.org/pdf/2610.31657v1",
      "published_date": "2026-10-23"
    }
  ],
  "run_these_manifests": {
    "title": "Run these Manifests yourself",
    "summary": "Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.",
    "manifest_ids": [
      "1792735075842543379",
      "1792737665293761959",
      "1792737665293123822",
      "1792737665293214113"
    ],
    "steps": [
      {
        "n": 1,
        "title": "Get a Synorb Key",
        "text": "Click Generate Free Agent Key at https://synorb.com/connect/start, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{\"client\":\"your-agent-name\"}`. The key is the `api_key` field of the answer. Save the key. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; 3 of them are starter Beacons."
      },
      {
        "n": 2,
        "title": "Read each cited Manifest by id",
        "text": "REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief."
      },
      {
        "n": 3,
        "title": "Check the claim",
        "text": "Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them."
      }
    ],
    "mcp_url": "https://mcp.synorb.com/mcp",
    "connect_url": "https://synorb.com/connect/start",
    "request": {
      "method": "GET",
      "url_template": "https://api.synorb.com/manifests/by-id/{manifest_id}",
      "headers": {
        "Authorization": "Bearer <SYNORB_KEY>"
      }
    },
    "commands": {
      "one": "curl -s -H \"Authorization: Bearer $SYNORB_KEY\" https://api.synorb.com/manifests/by-id/1792735075842543379",
      "all": "for id in 1792735075842543379 1792737665293761959 1792737665293123822 1792737665293214113; do\n  curl -s -H \"Authorization: Bearer $SYNORB_KEY\" \"https://api.synorb.com/manifests/by-id/$id\"\ndone"
    }
  },
  "links": {
    "html": "https://hangingcontext.com/stories/2026-10-24/llm-judges-unstable/",
    "json": "https://hangingcontext.com/stories/2026-10-24/llm-judges-unstable/index.json",
    "markdown": "https://hangingcontext.com/stories/2026-10-24/llm-judges-unstable/index.md",
    "newsroom": "https://hangingcontext.com/n/artificial-intelligence/",
    "edition": "https://hangingcontext.com/editions/2026-10-24T00/"
  },
  "related": [
    {
      "url": "https://hangingcontext.com/stories/2026-10-24/hybrid-retrieval-fusion-tax/",
      "headline": "Hybrid retrieval has a fusion tax: combining retrievers can make results worse",
      "slug": "hybrid-retrieval-fusion-tax"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-10-24/reward-stack-rewritten/",
      "headline": "Reward hacking gets four fixes at once: rubric dropout, twin-world credit, radial best-of-N",
      "slug": "reward-stack-rewritten"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-10-24/reasoning-markers-measured/",
      "headline": "The wait marker in reasoning models was inherited from training data, not invented by RL",
      "slug": "reasoning-markers-measured"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-10-24/moe-serving-gets-cheaper/",
      "headline": "The MoE tax is shrinking: four preprints attack sparse-model inference costs",
      "slug": "moe-serving-gets-cheaper"
    }
  ],
  "sponsor": {
    "label": "From Synorb",
    "title": "Ask what Synorb covers",
    "text": "The MCP tool synorb-stream-search answers questions like \"do you have this publisher?\" without using any Manifest allowance.",
    "cta": {
      "label": "Connect and ask",
      "url": "https://synorb.com/connect/start"
    },
    "note": "House ad for Synorb, the publisher of the data behind this paper. Not a news item."
  },
  "disclosure": "Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.",
  "reuse": {
    "version": "1",
    "summary": "Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product.",
    "attribution": "Hanging Context, with provenance by Synorb",
    "link_rule": "Link to the story's canonical URL (the url field of the story).",
    "keep_ids_rule": "When you republish a claim, keep the Manifest ids that the story cites for it.",
    "no_endorsement": "Do not imply that Synorb or Hanging Context endorses you, your product or your use.",
    "license_url": "https://hangingcontext.com/terms/reuse/",
    "machine_url": "https://hangingcontext.com/.well-known/reuse.json",
    "canonical_url": "https://hangingcontext.com/stories/2026-10-24/llm-judges-unstable/"
  }
}