LLM judges change verdicts with the prompting strategy — so the field moves to mechanical verification

The judge is the variable

If you evaluate models with models, you have probably done prompt engineering on the judge. A multilingual study by Asal Mehrabi, Mohammad Aliannejadi and Clemencia Siro, posted as a preprint this week, suggests that work may not travel: prompting strategies like chain-of-thought and role conditioning do not transfer reliably across judge models or languages, and the choice of judge model is the dominant factor in evaluation outcomes, overriding the choice of prompting strategy. You are not tuning the prompt so much as tuning to the judge. m:…3379

The instability gets worse where it matters most for global deployments: evaluation stability degrades significantly in low-resource languages. That means aggregate multilingual scores can hide per-language failures — a single average number may be averaging away exactly the cases you care about. m:…3379

Replace the verdict with a machine check

One response is to stop asking the model to judge at all and check the work mechanically. Anna Mei and Michael R. Douglas take this route for autoformalization — translating mathematics into Lean proofs — where they combine LLM-based semantic review with mechanical numerical and vacuity checks. The hybrid approach reduces the accepted-error rate in Lean formalizations compared to LLM-only verification. Alongside it, the authors built the Formalized Library of Mathematical Functions (FLMF), a partial formalization of NIST DLMF content, as a benchmark for evaluating autoformalization methods. m:…1959

The lesson generalizes past mathematics: anywhere an LLM's verdict can be checked by computation rather than by another LLM, the mechanical check is the cheaper source of truth. High-assurance agentic pipelines should be treating LLM judges as the first pass and the mechanical check as the verdict. m:…1959

Tune the evaluation, don't write it

A second response keeps the LLM in the loop but removes the prompt-engineering tax on evaluation itself. In a preprint, Selen Erkan and colleagues propose soft-prompt tuning: instead of hand-crafting prompts and parsers for each benchmark, they adapt models to benchmark formats through tuned soft prompts. Format-following saturates within 150 steps — about 1,200 samples — while knowledge accuracy is preserved, and the method matches strong evaluation-harness baselines without any manual benchmark-specific prompt or parser design. It also improves the evaluation signal on long-context datasets and instruction-tuned variants with different output styles. m:…3822

The throughline is familiar: evaluation pipelines keep failing on the same brittleness — the prompt is a variable nobody controls. Tuning the evaluation prompt rather than hand-writing it is the obvious next step, and it is a small one compared to retraining a judge. m:…3822

Price the mistakes instead

A third paper asks the cost question directly. In a preprint on LLM-based IT ticket classification, Sachin Gupta adds abstention to the picture: the model can hand a ticket to a human when it is unsure, and the routing policy is calibrated on actual costs. The measured result is striking — TF-IDF with logistic regression at a 0.60 threshold reduced test cost by 38.1% and wrong routes from 15.3% to 3.5%, and calibration-selected policies beat always-route and always-defer strategies in 14 of 16 tested configurations. The paper also finds that isotonic recalibration lowers expected calibration error without reliably improving discrimination: the standard calibration metric is not the operational one. m:…4113

The quiet punchline: on a cost-sensitive task, a linear model with a threshold outperformed the instinct to throw a bigger LLM at the problem. The arc of the week is the same across all four papers — the judge is unreliable, so verify mechanically, tune the evaluation prompts, or price the mistakes. The era of trusting a single LLM's verdict is ending; the era of instrumenting it is here. m:…3379 m:…1959 m:…3822 m:…4113