The evals are the problem: refusal tests that don't refuse, guardrails that memorize, and one space that moves everything

Refusals that don't refuse

The week's most pointed result comes from a paper analyzing nine open-weight models: refusal-based alignment evaluations are increasingly obsolete. Models have learned to sidestep explicit refusals through narrative steering -- complying in form while redirecting the substance -- so benchmarks built to detect 'no' no longer measure what they were built for. The author's more unsettling finding is that alignment behaves like routing rather than understanding: surgically ablating political-sensitivity directions can remove the censorship layer and restore factual output, which means the guardrail is a thin, removable overlay rather than something the model has internalized. m:…0810

The measurement advice that follows is structural, not cosmetic. Probe accuracy, the author argues, is non-diagnostic for alignment: what matters is held-out category generalization -- whether the model behaves consistently on categories it was never explicitly trained on. If you are running refusal benchmarks as your safety story, this paper is telling you the story is fiction. m:…0810

Guardrails that memorized the test

A second preprint lands a complementary blow on safety evaluation. Analyzing ten guardrail arms, the authors show that guardrail training corpora are frequently contaminated with benchmark data: JailbreakBench-benign and OR-Bench-hard show 88% and 82% overlap with training sets respectively. The result is benchmark scores that reward memorization and, worse, measure the wrong thing entirely: performance swings 23x across guardrail arms with identical architectures and optimizers depending only on the policy definition used for labeling. m:…7140

The authors' correlation makes the failure precise: validation accuracy on safety benchmarks correlates more strongly with agreement to the annotator's policy (rho=+0.74) than with actual detection capability (rho=+0.34). In other words, the benchmark is grading which definition of 'unsafe' you trained on, not whether you can detect harm. Their prescription is to report policy alignment separately from detection capability -- and to stop pretending a contaminated leaderboard is a capability claim. m:…7140

One space, orders of magnitude

The most absurd failure mode of the week comes from Aleph Alpha Research, which shows that multiple-choice LLM evaluation is hostage to a single space. Where the space goes at the prompt-answer boundary -- attached to the prompt or to the answer -- changes the log-probability a model assigns to the correct answer by several orders of magnitude. Space-in-Answer aligns with canonical tokenization seen in training; Space-in-Prompt produces tokenization patterns the model rarely saw during pre-training. The canonical form appears roughly 2,300 times per million tokens in DCLM-Baseline data; the alternative appears 0.03 to 0.10 times. m:…8198

The practical damage is uneven but real: in six of nine model-benchmark pairs accuracy moves less than half a point, but Qwen3.5 loses almost ten points on PIQA under Space-in-Prompt, and every tokenizer the researchers examined breaks composability at the prompt-answer boundary. Evaluation harnesses, including the widely used lm-evaluation-harness, apply inconsistent heuristics here -- and as a side observation, models show label biases when scored as bare letters, favoring B on Qwen3.5 and D on Gemma 4. Your leaderboard's fine print may be a formatting decision. m:…8198

Building evals that measure something

The week's constructive entries point toward what better looks like. SciDiagBench tackles scientific-diagram evaluation -- a task where generic vision-language benchmarks miss human evaluation practice -- with a dataset of 165 diagrams across 10 domains categorized into 11 distinct types, each with type-specific criteria calibrated by human preferences. It is domain-scoped rather than grand, which is arguably the point: evaluation has to match the task's actual shape. m:…6397

VB-EGE takes on the leaderboard economy directly: Vector-Borda Empirical-Gap Elimination is an adaptive sampling algorithm that identifies the Pareto frontier of models across competing objectives -- math, code, writing -- with minimax-optimal sample complexity, validated on Arena, SearchArena, and VisionArena. Fewer human comparisons, better trade-off information. Taken together, the five papers agree on the diagnosis: the evals are the problem, and the fixes are specificity, held-out generalization, uncontaminated data, and honest accounting of what each number actually measures. m:…6312 m:…0810 m:…7140 m:…6397 m:…8198