When alignment works in one setting and fails in the next

Alignment that travels badly

In a preprint, researchers report that probing can locate small sets of attention heads tied to specific human values and stylistic preferences, and that steering those activations while a model runs can align it with one value or with several combined values, without changing the model's parameters. This suggests alignment could become something adjusted per request rather than fixed once at training time, which would matter for products serving users with different values. But the same batch of preprints shows how narrow that control can be: a method that steers a model well in a clean test may not steer it well once the input is attacked, the values compete, or the training signal itself has been tampered with. m:…9860

The fragility is measurable. In a preprint on activation steering, researchers report that robustness falls by up to 64% when inputs are changed with adversarial text perturbations, and that the best layer to steer from shifts by up to 17 positions under those perturbations. Pulling steering vectors from the perturbed inputs recovers some steerability, the authors report, but it does not find the best layer again. This suggests steering is not a stable dial a builder can set once and trust; its best setting moves with the input, and an attacker who changes the wording can move it a long way. m:…3703

Images show the same weakness

The problem is not limited to text models. In a preprint on text-to-image generation, researchers describe a method called BadReward that enables indirect clean-label poisoning of preference-based training, using image pairs that look harmless but line up with a malicious target concept in the reward model's feature space. The key point for builders is that the poisoned examples do not look poisoned: they pass a visual check while still bending what the reward model learns to prefer. Online feedback loops make this worse, the authors report, amplifying reward corruption compared with offline preference methods, because each round of feedback can feed the corrupted signal back in. m:…1658

Put together, these preprints point to the same gap from opposite sides. Value localization and activation steering give builders finer control over what a model does, but that control rests on attention heads and reward signals that can be shifted or poisoned without any obvious warning. This suggests the practical question is not whether a steering method works on a clean benchmark, but whether its control survives hostile inputs and a training pipeline that someone may have tampered with. A team that deploys steering without testing both conditions is, on this evidence, testing only the easy half. m:…9860 m:…3703 m:…1658

Training order helps, but only so far

A fourth preprint offers a partial answer on the training side. Its authors report that a staged method called Staged-Competence reduces harmful responses outside the training distribution by 13.6 percentage points compared with standard preference training, and reduces jailbreak success across five attacks by 16.5 percentage points, while reaching baseline safety performance with only 75% of the training data. The authors also report that standard preference datasets contain inconsistencies, which their PiKacHHU dataset is designed to mitigate. This suggests the order and cleanliness of training data matter as much as the alignment technique applied on top of it, and that a smaller, better-ordered dataset can beat a larger, messier one on safety. m:…9623

What this means for builders

For a founder deciding how to ship alignment, the four preprints together argue for treating alignment as a system to be stress-tested, not a property a model either has or lacks. Targeted removal of value-sensitive attention heads gave causal evidence of their role in model behaviour, the value-localization authors report, which means those components can be audited rather than merely trusted. This suggests a sensible bar before deployment: test steering under adversarial wording, test preference data for poisoning, and test safety outside the training distribution. A method that passes only its own clean benchmark has not yet shown it can hold up where users, attackers and competing values meet. m:…9860