Reasoning Models Are Learning to Ask Better Questions Before They Answer

Long answers from self-made tasks

In a preprint, LongZero enables large language models to self-improve long-context reasoning by synthesising structured multi-hop and retrieval tasks. It synthesises questions and answers with supporting evidence in a compact structured form before expanding the evidence into a long multi-document context, and it trains models using GRPO without external data. m:…1862

The reported result is a rise in average performance for Qwen3-4B-Thinking-2507 across five long-context benchmarks, from 54.7 to 57.4, and LongZero is reported to outperform reinforcement learning on data synthesised from external corpora. That comparison matters because human annotation for long-context reasoning data is expensive, and existing synthesis methods depend on external corpora or stronger teacher models. m:…1862

The practical claim is a scalable path: if a model can generate useful long-context tasks for itself, teams are less dependent on scarce or expensive external data to improve reasoning over long documents. The preprint does not claim the problem is solved, only that self-generated tasks can move the average on the five benchmarks tested. m:…1862

Reward the information, not just the answer

In a preprint, a second method tackles a related weakness: large language models struggle to adaptively gather information in experimental design tasks. The authors propose amortising sequential Bayesian experimental design into language models by training reasoning with information-gain rewards, so the model learns to choose questions that reduce uncertainty. m:…3566

The method is reported to outperform inference-time and training-time Bayesian experimental design baselines at lower computational cost, and to improve information acquisition and task performance on the Twenty Questions task. It also generalises to unseen categories, and shows positive transfer to the Medi-Q clinical dialogue benchmark, which suggests the skill of asking well transfers beyond the game it was trained on. m:…3566

The study also investigates design choices including generative-model constructions, reward estimation, hypothesis-space specification and multi-step objectives, with the experimental generative model specified independently of the language model's own beliefs. That separation is part of the method: the model is not simply rewarded for confirming what it already believes. m:…3566

Do you still need the supervised step

Patronus AI built an internal post-training stack to quantify the value of training tasks, automating five stages covering training, checkpointing, endpoint deployment, evaluation and leaderboard updates. The platform snapshots code and records git commits for reproducibility, and the team audited 50 passing runs to verify no reward hacking occurred. m:…0759

On GitHub tasks, supervised fine-tuning then reinforcement learning improved performance from 35.6 percent to 47.8 percent, while reinforcement learning alone improved it to 45.4 percent. On Unreal Engine tasks, supervised fine-tuning then reinforcement learning improved performance from 64.3 percent to 71.5 percent, while reinforcement learning alone improved it to 73.1 percent. The team concludes that reinforcement learning alone performed about as well as supervised fine-tuning then reinforcement learning. m:…0759

The experiments used Qwen3.8-27B as the base model, with Kimi-K3 as the expert model for supervised fine-tuning data generation, and GitHub reinforcement learning performance was still rising at step 60. That last detail is a warning against reading the comparison as final: the recipe that looks tied today might separate with longer training. m:…0759

Looking ahead, the team plans to experiment with 256,000-token context windows for Unreal tasks, explore partial credit rewards and adaptive task choice to optimise compute, and build online evaluation for model checkpoints and a unified evaluation harness for agentic benchmarks. It is also exploring rewarding individual actions in reinforcement learning, and plans a data flywheel to store and analyse trajectories. m:…0759