RL forgets, adds little over good fine-tuning — and still gets better at the hard parts

RL forgets

Reinforcement learning is often assumed to be immune to catastrophic forgetting during continual post-training. A preprint this week shows it is not. Standard RL methods suffer from significant behavioral drift when vision-language models are continually post-trained, losing earlier capabilities as they adapt to new ones. The authors introduce the Multimodal Reasoning Continual Learning benchmark, MRCL, a diverse evaluation suite for multimodal reasoning tasks in continual learning, to quantify the failure, and propose Continual Policy Optimization, or CPO, which uses a prior-task behavioral KL objective and a local Fisher surrogate to preserve historical capabilities. The claim is that forgetting can be reduced with negligible overhead, which matters for teams maintaining evolving multimodal models in production. m:…9633

A second preprint asks a blunter question: how much does verifier-based RL add on top of good supervised fine-tuning? On a new deterministic benchmark called gpu-train-debug, supervised fine-tuning accounted for the majority of performance gains. Verifier-based RL with GRPO added no detectable in-distribution improvement and only modest gains under distribution shift. The authors' conclusion is that for certain tasks the marginal utility of an RL stage may be lower than commonly assumed, which changes how training pipelines should allocate resources. m:…0961

Better methods for harder problems

The same week also shows RL improving where the problem is genuinely hard. RLVR squared, a ranking-based approach to reinforcement learning with verifiable rewards, replaces linear scalar aggregation of rubric scores with latent utility recovery from within-group ordinal outcomes. Linear aggregation assumes heterogeneous criteria can be compared on one scale, which the authors argue is unreliable. The method consistently outperformed representative rubric-based baselines across 16 benchmarks and three model scales, and it supports objective-preserving attribute adjustment, letting auxiliary attributes influence training without direct reward weighting. m:…9632

For interactive agents, a tree-structured method called TSBPO tackles the credit assignment problem: terminal feedback tells an agent whether it succeeded, not which step mattered. TSBPO uses a fine-grained budget allocator and tree-aware step-level credit assignment, and in a preprint it reached 97.79% seen and 96.64% unseen stochastic success on ALFWorld using a Qwen2.5-1.5B actor, and 81.07% per-episode success on WebShop. Removing the observation-group step term dropped WebShop success from 81.07% to 75.93%, which shows the credit assignment term is doing real work. m:…0962

The side effect: models stop speaking English

There is also a side effect worth naming. Patronus AI analyzed open-source reasoning models across coding tasks and found that 2026 releases have significantly reduced filler words in their chain-of-thought reasoning. Kimi K3.1 Pro is now the leanest at 9.8% filler, while Qwen3.8 Flash remains the wordiest at 23.7%. Human-written bug reports, for comparison, contain about 20.8% filler. But shorter wording did not mean fewer tokens: DeepSeek's newer model uses 2.5 times the reasoning tokens of its predecessor. m:…9002

Patronus hypothesizes the shift is driven by reinforcement learning penalties on response length, distillation from terse teacher models such as GPT-6 Astra, or adaptation to agentic output requirements. Agent harnesses can also influence how verbose model reasoning is, which means the same model can sound different in different tools. Outcome-only RL rewards can also lead models to develop private shorthand, which is the same forgetting problem from the first section, wearing a different costume: the model optimizes the reward and loses the readable reasoning along the way. m:…9002