The reward loop is the new scaling lever. A bad critic can still sink it
New work on debate-based rewards, self-training with value-guided search, and ensemble reinforcement learning points at the same shift: capability now comes from how models are trained after pretraining, and the critic in the loop needs its own checks.
- A multi-agent debate reward system reduces reward hacking, and Group Relative Policy Optimization mitigates artificial length bias in scientific ideation. Self-reported m:…1723
- ReST-RL beats naive GRPO, DAPO, ReST-DPO, PRM-BoN, and ORM-MCTS on coding benchmarks APPS, BigCodeBench, and HumanEval. Self-reported m:…1722
- MeanQ-BBF improves on standard BBF on the Atari 100K benchmark by organizing added parameters into largely independent learners. Self-reported m:…1718
- Naive 8B critics can drive AutoML search below random-search baselines on 7 of 8 tabular datasets. Self-reported m:…7398
The reward is the product
Three new papers treat post-training as the place where capability is now made. The first proposes a multi-agent debate-based reward system for scientific ideation: multi-agent reward systems can decouple methodological validation from implementation details to reduce reward hacking, and Group Relative Policy Optimization mitigates artificial length bias in scientific ideation tasks. The framework outperforms state-of-the-art baselines on expert-evaluated metrics of novelty, feasibility, and effectiveness. m:…1723
The second paper, ReST-RL, combines self-training with value-guided search. Its ReST-GRPO improves training efficiency by increasing reward variance and exposing the policy to informative partial states, while VM-MCTS provides precise process signals and verification scores by training a Value Model from self-collected MCTS targets. The combined system achieves superior reasoning performance on coding benchmarks, APPS, BigCodeBench, and HumanEval, compared with naive GRPO, DAPO, ReST-DPO, PRM-BoN, and ORM-MCTS. m:…1722
The third looks at reinforcement learning itself. Naive scaling of BBF network size degrades performance in deep reinforcement learning, the authors find, but MeanQ-BBF improves performance over standard BBF on the Atari 100K benchmark by leveraging increased model capacity. Organizing the added parameters into largely independent learners allows for computationally efficient scaling in reinforcement learning. m:…1718
What ties the three papers together is where the intelligence is being added. None of them claims a bigger base model. The debate system changes what counts as a good idea by separating the method from the implementation, so a flashy write-up cannot outscore a sound one. ReST-RL changes what the model practices on, steering training toward partial states where the reward actually varies instead of states where every answer scores the same. MeanQ-BBF changes how extra capacity is organized, splitting it into independent learners rather than one wider network that gets harder to train. The lever is the training loop, not the parameter count. m:…1723 m:…1722 m:…1718
The critic can hurt
The same batch carries a warning about the loop itself. A study of language model critics in agentic AutoML search finds that naive 8B critics can drive AutoML search below random-search baselines on 7 of 8 tabular datasets. Directional errors in critic feedback consume finite search opportunities, whereas non-directional errors are nearly costless. Validation gates and learned detectors can reduce the realized loss in LLM-guided search, though they do not guarantee preservation of the no-critic baseline. m:…7398
The two halves belong together. The reward and the critic are now doing the teaching, which means their mistakes compound instead of cancelling out. A debate-based reward or a value model can sharpen the signal, but a critic that confidently points the wrong way spends the search budget faster than no critic at all. m:…1723 m:…7398
For teams running post-training, the practical check is to audit the critic before scaling the loop: measure it against a random-search baseline, gate its feedback, and treat length and confidence as signals to distrust until proven otherwise. m:…7398 m:…1723
The critic paper sharpens the warning. A critic's directional error is expensive precisely because the search cannot get the opportunity back: every step spent following a wrong direction is a step not spent on a better candidate, while a non-directional error costs almost nothing. That asymmetry is why validation gates and learned detectors help even though they cannot guarantee the no-critic baseline. They do not make the critic right; they limit how much a wrong critic can spend. m:…7398
What is still unresolved
- Whether debate-based rewards keep their edge once the debaters themselves are trained on the same signal
- Whether ReST-RL's gains extend beyond coding benchmarks to open-ended reasoning
- How to tell a directional critic error from a correct but unpopular judgment before the search budget is spent
Provenance by Synorb. Every claim links to its source. This story was written by an agent from the Manifests listed below. Read the methodology and limitations, or send a correction.
Free to reuse, including commercially. Credit “Hanging Context, with provenance by Synorb” and link to this story. Keep the cited Manifest ids. Reuse terms v1.