Five ways to train agents dropped this week; two beat GRPO at the same GPU budget

The bottleneck everyone is attacking

Five preprints posted this week circle the same problem: training agents with reinforcement learning is expensive because the reward usually arrives only at the end. When a multi-turn agent finally succeeds or fails, the training signal says almost nothing about which of its fifty steps mattered. The DeltaTTG authors call this out directly — sparse terminal rewards give limited information about individual steps — and the TPPO authors report that standard PPO critics often fail to track how value evolves along a trajectory for the same reason. m:…9134 m:…2157

The RSIPO authors frame it as a credit-assignment problem for long-horizon tasks. Their fix on the teacher side is empirical Bayes shrinkage — a mechanism to calibrate how much the student trusts the teacher — which suggests the authors see teacher guidance as something to discount rather than follow blindly. m:…5097

Two beat GRPO at the same budget

The headline results belong to RSIPO and DeltaTTG, and the reason they matter is the cost line, not just the score line. In a preprint, the RSIPO authors report 98.6% success on ALFWorld with Qwen2.5-7B — 17.4 percentage points above standard GRPO — by letting agentic skills and the policy model co-evolve: the policy extracts skills from completed trajectories and replays them to get better at generating skills. It is a self-improvement loop with no external teacher in the critical path. m:…5097

DeltaTTG takes a different, almost embarrassingly simple route. In a preprint, the authors introduce a Token-to-Go meter — a learned estimate of how many tokens remain until task completion — and use it as a cheap, reusable progress signal to redistribute outcome-derived advantages across steps without touching the rewards. The reported gains are 5.7 points on Qwen3-4B and 7.2 points on Llama-3.1-8B-Instruct over GRPO, across ALFWorld, PaperSearchQA and BFCLv3, with actor-training GPU-hours staying close to GRPO's. A method that beats the baseline at the same compute is the one to replicate first. m:…9134

The rest of the wave

TPPO goes after the critic rather than the advantage signal. In a preprint, the authors argue that PPO's value estimates drift along a trajectory, then fix it with a cumulative value parameterization built on a Bayesian decomposition in log-odds space. They report 7.0 and 8.4 point gains on mathematical reasoning for Qwen3-4B-Base and Qwen3-8B-Base, plus better out-of-distribution generalization. If your agents do math, code, or anything with verifiable intermediate steps, the critic is where your PPO run is probably leaking performance. m:…2157

Ares zooms out to the optimizer itself. In a preprint, the authors report a genuinely new observation: under reinforcement learning with verifiable rewards, gradient energy concentrates into a shrinking effective subspace as training proceeds — something they did not observe under supervised fine-tuning, and which disappears when the verifiable reward is swapped for a random one. Their adaptive-rank optimizer, Ares, rides that geometry by adjusting its update rank to the gradient's spectral energy distribution, and they report accuracy gains over standard optimizers across model scales. The paper notes most RLVR still runs on optimizers inherited from teacher-forcing; the optimizer deserves its own design pass. m:…2525

SALD is the odd one out — it targets diffusion models, not language agents — but it rhymes with the week's theme of doing more with less. In a preprint, the authors describe a self-referenced training framework that evaluates each image-caption pair at two noise levels with the same model, using the difference to weight training adaptively. They report better generation quality with no teacher network, no extra parameters, and no change to inference. m:…4071

What to actually do with this

Treat every number here as a preprint claim, not a settled result — none of these papers are peer-reviewed, and the benchmarks (ALFWorld, BFCLv3, math reasoning sets) are narrow slices of real agent work. The honest move is to replicate DeltaTTG's progress-meter idea and RSIPO's skill-replay loop on your own task distribution before believing the point deltas. m:…9134 m:…5097

The deeper signal is architectural: credit assignment is getting its own machinery — progress meters, skill libraries, better critics, geometry-aware optimizers — instead of being left to the reward function. If you maintain an agent training stack, this suggests the coming stretch of agent-RL work could look like the optimizer wars of the 2010s, but for the RL loop around agents. m:…2525 m:…2157