Training cost is now search cost: configs, rollouts, data mixes and optimisers all get cheaper searches

Stop paying for a full run to test one idea

In a preprint, researchers introduce JoBS, a multi-fidelity Bayesian optimisation method for co-optimising LLM training configurations, including data and model hyperparameters, without requiring a full training run for every candidate. The method uses a scaling-law-inspired performance predictor to estimate fully trained performance from partial training steps, so more of the configuration space can be explored within a fixed budget. The authors report that JoBS outperforms independent data and model optimisation and existing multi-fidelity Bayesian optimisation baselines across a diverse set of downstream tasks. The decision it supports is mundane and valuable: a team with a fixed training budget should spend it searching jointly over data and model settings, guided by early signal, rather than tuning each side separately with full runs. m:…0952

A second preprint applies the same logic inside reinforcement learning from verifiable rewards. HORA replaces uniform rollout allocation with hit-utility allocation, shifting rollout budgets to prompts more likely to yield correct trajectories, and is reported to improve Pass@K over compute-matched GRPO while remaining drop-in compatible with group-based estimators such as GRPO and RLOO. Notably, the uniform prior used by HORA is reported to be competitive with five prompt-conditioned learned-prior alternatives, which means the gain comes from the allocation rule rather than from a learned model of prompt difficulty. For a team scaling reasoning training, that is a useful distinction: the cheapest improvement may be spending the same rollouts differently, not buying more of them. m:…5182

Simulate the mixture instead of retraining for it

A third preprint, on data mixing across the whole LLM lifecycle, proposes OP-Mix, which simulates candidate mixtures by interpolating between low-rank adapters trained on the current model, rather than training separate proxy models or retraining for each candidate. The authors report a 6.3% improvement in pretraining perplexity compared with training without mixing, and compute reductions for continual learning of 66% compared with retraining and 95% compared with on-policy distillation. The practical claim is that mixture search can be grounded in the model's actual learning dynamics, through adapters on the model itself, instead of in a smaller stand-in whose preferences may not transfer. m:…1362

A fourth preprint attacks the optimiser itself. LionMuon interleaves the Muon optimiser, which is powerful but expensive in Newton-Schulz iterations and all-reduce communication, with the cheaper Lion sign-based update, performing a Muon step every P iterations. The authors report up to 30% lower wall-clock training time on 4-GPU PCIe systems compared with Muon, lower loss than Muon, AdamW, Lion and Signum on 124-million and 355-million parameter models, and a single optimiser state buffer per matrix, half the memory requirement of AdamW. The pattern is the same as in the other three papers: keep the expensive step, but pay for it less often, and let a cheap step carry the intervening work. m:…7515

Budget for search, not just for the final run

Taken together, these preprints describe a training budget that is mostly search: searching configurations, searching rollouts, searching mixtures and searching update directions. Each paper replaces a full-price search with a cheaper proxy, a partial run, an adapter interpolation, a reallocated rollout budget, or an interleaved optimiser, and each reports that the cheaper search preserves most or all of the benefit. The implication for a team planning a training run is to budget explicitly for search, and to ask of every proposed method whether its savings survive at the team's own scale, hardware and data mix, because all four results are preprints tested in the authors' setups, not guarantees for a different setup. m:…0952 m:…5182 m:…1362 m:…7515