The refusal test: the same biology question gets refused, answered, or ignored depending on the model

The benchmark

Benchling built BenchBench-Refusals to answer a narrow question: can frontier models tell a harmful biology request from a benign one? It tested 160 prompts across common biology tasks, divided into benign, pseudo-hazardous, and hazardous risk levels. The headline finding is that safety performance varies sharply by provider, and that general capability is not a reliable indicator of safety calibration. As Benchling's Nicholas Larus-Stone put it: "Better models are not necessarily better calibrated." m:…6917

OpenAI's models distinguished between benign and harmful biology requests reliably, and refused at roughly the same rates regardless of model size. Anthropic's models were reliable when enrolled in the company's Life Sciences Verification Program, but without it, models such as Fable 5.1 and Opus 5.5 refused many routine, benign tasks. Gemini and Grok sat in the middle. The worst result belonged to open-weight models, which Benchling found rarely refuse any requests, including harmful ones. Astra, an open model, had a refusal rate on par with models that perform much worse on biology. m:…6917

The benchmarks are confounded

A preprint from a cheminformatics researcher argues the chemistry safety benchmarks themselves need auditing. Molecular identifiers significantly influence refusal rates, and system prompts skew assessments, so existing benchmarks such as ChemSafetyBench and DualChem are confounded by variables that have nothing to do with safety. The paper also found that excluding tool failures, such as web access errors, changes the apparent safety benefit of language models. The proposed fix is a controlled framework that holds the chemical question constant while varying identifiers and inputs, so a refusal can be traced to safety judgment rather than a formatting accident. m:…7762

A second preprint, a perspective on LLMs in chemistry, adds a different warning: the models generate fluent text that is often factually incorrect, because chemical information is dense with symbols, units, and stoichiometry. Its authors propose a taxonomy of chemical hallucination types and a verification framework that wraps LLM outputs in cheminformatics libraries, databases, and physical measurement protocols. The practical advice is to treat the model as a fallible assistant, not as a source of chemical truth. A related study of four open-source LLMs for molecular generation found quantization can raise throughput up to 24-fold, but only Gemma-4-31B kept its adherence to reference chemical spaces across all quantization levels, while GPT-OSS-120B lost reference-guided generation at Q4. m:…3847 m:…8474

Values are one fine-tune away

Nous Research's Arete-27B shows how quickly a model's values can be rewritten. The team post-trained Alibaba's Qwen3.8-27B with 52.8 million tokens of supervision to reinforce Western values, and reported that propaganda-laden answers fell from 22% to 1% on an 803-prompt evaluation spanning seven value categories, while the fact score rose from 2.32 to 2.98 out of 5. General agentic capability was retained: Arete scored 65.60% on the Hermes Bench, against 65.77% for the base Qwen model. Nous's own summary of the lesson is blunt: "The values in any open model are only ever one fine-tune away." m:…4714

Read together, the three threads point in one direction. Refusal is now a measured behavior, not a vibe, and the measurements show it depends on the provider, the verification program, the prompt, and even the molecular identifier used in the question. Open-weight models that rarely refuse anything are the sharpest edge of that finding. And the Arete result shows the values layer is shallow enough to be retrained with tens of millions of tokens, which cuts both ways: it can be fixed, and it can be removed. m:…6917 m:…4714