Better reasoning is getting cheaper to train

GRPO catches PPO when the batch grows

Scale Labs Research ran an empirical study spanning more than 1,200 training runs to investigate the trade-off between PPO and GRPO, the two reinforcement learning methods most teams choose between when training reasoning models. PPO uses a separate critic network to estimate state values and lower gradient variance, while GRPO compares rewards across a group of trajectories sampled from the same prompt instead of using a critic. GRPO saves memory, is simpler to implement, and avoids bias from bootstrapping off an imperfect value function. m:…6184

The headline finding is about scale, not about which method is better in the abstract. PPO generally leads at small batch sizes, while GRPO catches up and often overtakes PPO as batch size grows, and more complex tasks and longer horizons push the crossover point to larger batch sizes. In math tasks, GRPO overtakes heavily bootstrapped PPO at batch size 1,024, and in terminal agent tasks, GRPO reaches parity with PPO by a batch size of 2,048. At small compute budgets, PPO's critic saves enough rollouts to justify its extra training cost, but at large compute budgets, GRPO can match PPO with less training compute. ScaleRL, the system behind the study, devotes 80% of its GPUs to generating rollouts and 20% to policy updates, which shows where the compute actually goes. m:…6184

No ground truth needed

A preprint on multi-agent reinforcement learning, Co-RL, attacks a different cost: labels. It introduces an approach that eliminates the need for ground-truth supervision by leveraging rewards generated by a diverse cohort of decoupled models, and it improves reasoning performance by 3.0-7.9% across seven benchmarks without ground-truth labels. The method addresses the common pitfalls of self-rewarding reinforcement learning, such as response homogenization and training collapse, by using heterogeneous model families and sizes to break self-reinforcing error loops. m:…6052

Increasing cohort diversity through heterogeneous model families reduces correlated errors and prevents training collapse in self-rewarding reinforcement learning, and Co-RL outperforms CoMAS by 4.0% and MM-UPT by 2.0% in reasoning tasks. The implication is that the expensive part of training reasoning models may not be the compute at all, but the human labels, and a diverse enough cohort can stand in for them. The preprint status matters here: the gains are the authors' own measurements, and the method's value depends on whether the diversity it needs is cheap to assemble. m:…6052

Fixing the critic, and skipping the tool call

A preprint on multi-task reinforcement learning, TOPPO, fixes a specific weakness in PPO. In multi-task settings, PPO suffers from critic-side gradient ill-conditioning, causing tail tasks to stall, and TOPPO introduces Critic Balancing modules to address it. TOPPO outperforms SAC and ARS-family methods on the Meta-World MT50 benchmark, and achieves final performance in fewer environment steps and with fewer parameters than SAC-family methods. For teams training one model across many tasks, the tail tasks are often the ones that matter in production, and they are the ones PPO was leaving behind. m:…1964

The fourth study is about inference cost, not training cost, and it is the most immediately usable. LLM agents often trigger unnecessary tool calls, increasing latency and API costs, and a preprint on tool use shows that tool necessity is linearly decodable from the pre-generation hidden states of LLMs with an AUROC of 0.89-0.96. The proposed Probe&Prefill method reduces tool calls by 38% with only 1.7% accuracy loss, outperforming existing prompt-only and reason-then-act baselines. Existing training-free baselines for tool-use control, such as Prompt-only and Reason-then-Act, suffer from either excessive accuracy loss or insufficient tool-call reduction, so the probe is not a marginal improvement on a solved problem. m:…6040

What changes for builders

Taken together, the four studies describe a training stack that is getting cheaper from several directions at once. GRPO removes the critic at scale, Co-RL removes the labels, TOPPO fixes the critic for the teams that keep it, and a probe removes more than a third of wasted tool calls at inference time. None of these is a new model, and none of them will show up on a leaderboard as a product launch. They are the kind of result that shows up instead in a training bill and in the number of rollouts a team can afford. m:…6184 m:…6052 m:…6040

The decision for a team training reasoning models is now a set of trade-offs with numbers attached: batch size determines whether GRPO or PPO wins, cohort diversity determines whether unsupervised rewards hold up, and a hidden-state probe determines whether tool calls are worth making. The numbers come from the authors' own experiments, and three of the four studies are preprints, so they should be treated as directions to test, not as settled facts. But the direction is consistent, and it points to reasoning getting cheaper to build. m:…6184 m:…6052