Models Know the Risk and Answer Anyway: The New Safety Gap in Chemistry and Robotics

The chemistry test

Scale AI says it launched DualChem to measure how frontier models handle requests that could support either legitimate scientific research or harmful activity, and the company says the benchmark was developed in collaboration with the NIST Center for Advancing Innovation and Standards for Super Intelligence, known as CAISSI. The benchmark consists of 600 expert-curated questions spanning multiple laboratory chemistry domains, and it measures whether a model provides the same answer to support either legitimate research or harmful activity. That design matters because a single request can look benign in isolation and dangerous in context, and a test that only counts refusals would miss the harder question of whether the model understands the dual-use nature of what it is being asked. m:…2455

The early results reported by Scale AI are uncomfortable. Frontier models demonstrated an average accuracy of about 50 percent on the dual-use chemistry questions, and models frequently identified questions as potentially dual-use but still provided substantive answers in most cases. Risk identification and refusal behaviour appeared only weakly connected, and model predictions for severity scores showed weak alignment with human judgments and significant variability. The company says refusal-based safety mechanisms alone may be insufficient for handling complex dual-use scenarios, a conclusion that shifts attention from whether a model says no to whether its judgment about severity and context is sound. m:…2455

A related finding sharpens the stakes. According to Scale AI, assistance from language models raised the performance of participants with bachelor's degrees in chemistry on dual-use tasks from about 32 percent to about 48 percent. That is a measure of capability uplift, not a claim that any particular harm followed, and it is the kind of result that makes dual-use evaluation different from ordinary capability testing. The question is not only whether a model can answer, but whom the answer helps and by how much, which is why the benchmark pairs accuracy with severity judgment rather than treating either alone as sufficient. m:…2455

The robot parallel

In a preprint, researchers studying household robot navigation report a strikingly similar pattern. They built OSMA-Bench to evaluate whether robot navigation policies adapt continuous motion to obstacle safety semantics, using spatially matched scenario sets that differ only in obstacle type. Vision-language-model-based policies and vision-language-action-based policies did not consistently adapt motion to obstacle safety, even with explicit safety prompts. Policies could correctly distinguish obstacles by safety relevance but failed to reflect that distinction in motion parameters such as clearance and distance-dependent speed and smoothness. m:…7306

The preprint also reports a partial remedy. Adding an explicit parameter-inference step improved adaptation for vision-language-model-based policies, which suggests the failure is not only about perception. The robot can see the difference between a fragile object and a sturdy one, in the terms of the benchmark, but the step that converts that perception into slower, wider, smoother movement is where the system breaks down. That distinction between recognising risk and acting on it is the same distinction DualChem exposes in chemistry, and it suggests safety work should be judged on behaviour under realistic conditions rather than on whether a system can label a hazard correctly. m:…7306

What constraints change

In a second preprint, researchers revisit constrained decoding, the technique of restricting what a language model is allowed to output, and challenge the assumption that it inherently degrades reasoning. They report that apparent performance degradation often stems from non-equivalent prompting and output formats rather than from the decoding process itself, and that the impact of a constraint depends on its type and on the output space it removes. Validity constraints improved reliability and extractability by eliminating invalid answers, while reasoning constraints could improve reasoning and accuracy when they enforced beneficial trajectories. The study evaluated constrained decoding across four model families and eleven open-source models. m:…0899

Read together, the three pieces point to a practical programme. First, measure behaviour, not just recognition, whether that means answers to dual-use chemistry questions or speed and clearance around obstacles. Second, treat the conversion step as a first-class design problem: the parameter inference that turns a safety judgment into motion, or the constraint that turns a policy into an allowed output. Third, be specific about which constraint or intervention is being tested, because a blanket claim that constraints help or hurt hides the differences between format, validity, and reasoning constraints that the preprint documents. m:…2455 m:…7306 m:…0899

What remains open

None of these results settles how a lab or a deployer should set thresholds in production. DualChem reports weak alignment between model severity scores and human judgments, which leaves open how much weight a deployment should put on a model severity score at all. The robot work shows an inference step can improve adaptation, but does not establish that the improvement holds across the full range of homes, obstacles, and hardware a product would meet. The constrained decoding work shows well-designed constraints can help, but its categories still need to be mapped onto the specific output spaces of real products, where a validity constraint that is harmless in one setting could block a legitimate answer in another. m:…2455 m:…7306 m:…0899