Who gets credit inside a chain of thought?

The credit problem, plainly

Reinforcement learning with verifiable rewards — RLVR — has a bookkeeping problem. The model generates a long chain of thought, gets a reward only at the end, and someone has to decide which tokens earned it. GRPO-style methods solve this by sampling groups of rollouts per prompt and assigning credit from group-relative outcomes — effective, but expensive. Three new preprints, all published attack the same question from three different angles: how to give the right tokens credit without paying for eight rollouts, reinforcing the wrong steps, or drowning in variance. m:…8075 m:…7997 m:…9697

HP-RLVR: one rollout instead of eight

HP-RLVR, from Wei Chen, Yubing Wu, Zhou Xu, Haoyue Zhang, Delu Zeng, Jiahao Wang, Junmei Yang, Pan Tiewen, Jingfeng Liu, Zhiheng Zhou, Bo Li, and Min Chen, proposes hindsight-privileged credit assignment: use hindsight context to assign token-level credit from a single rollout, with no group sampling required. The reported numbers are large — in the preprint, the method trains 1.21x, 1.46x, and 2.30x faster than GRPO on mathematics, code generation, and ALFWorld respectively, while using one-eighth the rollouts per prompt. It also reports eliminating the zero-gradient records that plague group-relative methods on ALFWorld: 43.2% (4B) and 49.0% (8B) of GRPO's gradient-norm records were zero, versus none for HP-RLVR. m:…8075

TACO and GGPO: fix the contamination, drop the weights

TACO — Tail-Aware Credit calibratiOn — from Lou and colleagues diagnoses a subtler failure — positive-credit contamination — where uniform credit assignment reinforces locally erroneous tail tokens inside trajectories that still end up correct. The fix uses the policy's own expectations to compute a tail-risk score, dynamically recalibrating each token's credit without any token-level error labels. In the preprint, the method improves training stability and performance across four LLMs and eight reasoning benchmarks compared with GRPO-style baselines. m:…7997

GGPO — Gibbs-Gap Policy Optimization — from Xuanchang Zhang and Tong Zhang takes on off-policy RL, where stale behavior policies force the use of learner-to-behavior importance weights that inject high variance into training. The method minimizes a log-mean-exp contrast of KL-adjusted rewards, eliminating those weights entirely. In the preprint, it outperforms GRPO and AGRO under training-inference mismatch and on fixed offline data — and the reference-refresh schedule turns out to be decisive: a five-benchmark average of 46.58 with dynamic refresh versus 32.96 with a fixed reference. m:…9697

For developers deciding what to do with three unreviewed preprints, the honest posture is watchful: all three papers appeared all benchmark against GRPO-style baselines, and the HP-RLVR and GGPO abstracts disclose no code repository for reproduction. That makes the near-term move tracking independent reproductions — and watching whether the three approaches (single-rollout sampling, tail-risk recalibration, weight-free off-policy updates) can be combined in one training loop rather than treated as rival forks. m:…8075 m:…7997 m:…9697

The three papers are really three diagnoses of the same disease — and that framing is the author's synthesis, not the papers' own claim. HP-RLVR says the cost of group sampling is the problem; TACO says the contamination of credit inside successful chains is the problem; GGPO says the variance of importance weights is the problem. For developers running reasoning-model training, the practical takeaway is that the GRPO default is getting crowded with challengers — each targeting a different bottleneck, none yet peer-reviewed, and no published code in the abstracts to kick the tires with. Treat all three results as preprint claims, not settled findings. m:…8075 m:…7997 m:…9697