Agentic RL grows up: verifiable rewards, async pipelines, rubric-guided research

Rewards you can check

Agentic RL has a credit-assignment problem: in an open-ended trajectory, which intermediate steps deserve the reward? Lean4AgenticRL, in a preprint, answers by grounding multi-level process rewards in a unified Lean4 formalization schema — the first framework to do so, to the authors' knowledge — with a workflow-stage-normalized GRPO algorithm providing the multi-level rewards. The framework first builds a problem-specific workflow system that balances the agent's own capability and controllability during execution, laying the foundation for verification. m:…2933

To keep this distinct from last edition's formal-verification story: the Lean4 machinery here is not verifying generated code. It is a formalization layer that makes intermediate RL rewards verifiable. And it scores: the LAR-27B model reaches 77.3% on TerminalBench-2.1 and 84.2% on SWE-Bench-Verified, with the model family averaging 9.89% over its base model. The LAR family also surpasses the second-best model of comparable size by 5.07%. m:…2933

No critics, no problem

TreeOPSD attacks the same long-horizon problem from the algorithm side: long-horizon reinforcement learning for language-model agents without critics. Terminal rewards alone make turn-level credit assignment sparse and high-variance, so the method treats each interaction turn as a tree node, branches alternative continuations from shared prefixes, and recursively backs terminal rewards up into implicit turn-level credit. The remedies it replaces — learned critics, process annotations, stronger external teachers — all increase training complexity. Each backed-up credit maps to a closed-form, bounded teacher distribution parameterized by the turn-level likelihood ratio relative to the frozen behavior policy. m:…3654

Under matched rollout budgets it outperforms GRPO, GSPO, and fixed-credit tree policy gradients — and lifts Qwen3-30B-A3B-Thinking-2507 by 4.97 points in the Deep Research setting. My reasoning: dropping the learned critic removes an entire failure surface from agentic training pipelines. m:…3654

Rubrics where verifiers don't exist

RubricEM handles the tasks that resist exact answer verifiers — long-form research. In a preprint, rubric-guided scaffolding teaches the agent to organize its trajectory into semantic stages, and Stage-Structured GRPO enables finer-grained credit assignment where no exact verifier exists. m:…4860

The resulting RubricEM-8B outperforms existing open models of larger sizes on four long-form research benchmarks. The pattern to watch: for research agents, structured human judgment — rubrics — is standing in for the math verifier. m:…4860

The production plumbing

LoongSage is the infrastructure paper: a production-grade, high-efficiency agentic RL framework for frontier LLMs with validated recipes. Its fully asynchronous RL pipeline eliminates synchronous idle bubbles; TransferMesh routes weights over the fastest available transport to cut trajectory staleness; and a Teacher Manager enables multi-teacher on-policy distillation while decoupling teacher-model scaling from GPU memory constraints. This is the unglamorous part that decides whether agentic RL runs in production or dies in a notebook. m:…2416

Know which tasks are hard before you roll

The last piece is diagnostic. A preprint on predicting task difficulty without rollouts finds token-level entropy is a useful predictive signal for task difficulty — letting teams estimate hardness without expensive rollouts. It also warns that AUC metrics can mask poor difficulty estimates, and that residuals between expected and observed difficulty expose environment flaws such as contamination and infeasibility. m:…0803

That last finding doubles as eval hygiene for anyone benchmarking agents: if your hard tasks are contaminated, your difficulty curve is fiction. Residuals between expected and observed difficulty are where you check. m:…0803