Training is learning to survive its own shortcuts

Distillation that tolerates cheap inference

On-policy distillation, where a student language model is trained on its own responses guided by a fixed teacher, turns out to be far more forgiving than reinforcement learning about the gap between training and inference. A preprint reports that on-policy distillation tolerates substantial mismatch without meaningful loss of training quality, remains stable under rollout configurations that destabilize RL methods, and on a 7B student model maintains training quality at over 200 times the mismatch of a standard BF16 engine. Quantized inference and asynchronous rollouts introduce that mismatch, because the rollout engine and the model being trained diverge, yet distillation keeps working. m:…7628

The practical payoff is speed and capacity. The authors introduce a lightweight admission check that screens rollout configurations against a calibrated budget without requiring a full training run, so teams can test a cheap rollout setup before committing to it. FP8 recipes provide capacity for at least 1.93 times as many KV-cache tokens as BF16, and combining FP8 with asynchronous rollouts reaches training quality 1.62 times sooner than synchronous BF16 training. One detail is worth remembering: the first layer of the model accounts for 93% to 99% of FP8 cache mismatch, linked to key-projection bias in DeepSeek student models. Precision loss is not spread evenly through the model. It concentrates in one place. m:…7628

Teach the agent to look, not to guess

Reframe, a two-stage post-training framework for agentic visual reasoning, treats observation states as objects of alignment. Its first stage, Ground-Drop-GRPO, trains specialists with evidence-aware rewards and stochastic removal of the original visual input, forcing the model to rely on tool-returned evidence rather than its own prior. Its second stage, Observation-Gated-MOPD, replaces deficient tool actions with canonical ones mid-rollout and resumes distillation from the repaired observation. The framework improves tool use and outperforms Mix-RL and vanilla MOPD across image and video benchmarks, the authors say, and it is built on the Qwen3.5-9B model. m:…7626

Neither stage requires a task-specific SFT cold start, which matters because cold-start data is often the most expensive part of post-training a visual agent. The underlying problem Reframe addresses is that visual agents learn to answer from their own output distribution rather than from what their tools actually returned. By removing the original visual input during training and repairing bad tool actions mid-rollout, the framework makes the observation the thing the model must align to. For teams building agents that use screenshots, cameras or document tools, that is the difference between an agent that looks and one that guesses. m:…7626

Stability from the architecture up

KiteNorm attacks stability at the architecture level. Its approach is to constrain hidden-state variance to prevent gradient vanishing and rank collapse in post-LayerNorm Transformers. The method outperforms pre-LayerNorm baselines across depth, width, batch size and training budget for decoder-only Transformers up to 1.3B parameters, the authors say. The comparison matters because pre-LayerNorm is the baseline the field actually uses, so beating it across depth, width, batch size and training budget is a broad claim, not a narrow one. m:…4179

TOPPO addresses a different stability failure, in multi-task reinforcement learning. PPO in multi-task settings suffers from critic-side gradient ill-conditioning, causing tail tasks to stall, and TOPPO implements critic balancing modules to resolve it. The method outperforms SAC and ARS-family methods on the Meta-World MT50 benchmark in both mean and tail-task performance, and achieves final performance in fewer environment steps and with fewer parameters than SAC-family methods. The result is notable because on-policy methods were widely considered inferior to off-policy standards for multi-task RL; TOPPO suggests the gap was a critic problem, not an on-policy problem. m:…1964

Alignment that does not drift

Proximal Preference Optimization rounds out the picture by combining the implicit reward formulation of DPO with the trust-region stability of PPO. The authors introduce two methods, P-squared-O and PKTO, which provide bounded policy divergence and monotonic improvement guarantees. P-squared-O introduces a clipped importance ratio into the DPO preference margin to enable reward-model-free alignment with bounded policy divergence, while PKTO incorporates prospect-theoretic utility margins and an adaptive baseline to manage asymmetric preference datasets. Both demonstrate improved stability under repeated dataset reuse compared with standard DPO and KTO, the authors say. m:…6037

The theme across all five preprints is that post-training is no longer about finding a cleverer loss function. It is about surviving the conditions of real training: quantized rollouts, asynchronous data, multi-task critics, repeated dataset reuse and agents that must trust their observations. Distillation tolerates mismatch that destabilizes RL, critic balancing unblocks multi-task PPO, and bounded-divergence alignment survives dataset reuse. The teams that internalize this shift will spend less time fighting their own training pipeline and more time on the model itself. m:…7628 m:…6037