LLM training breaks out of the single datacenter
Streaming DiLoCo cuts cross-slice communication by over 60x, and new RL frameworks train agents across sites without stalling. Location stops being the constraint.
- Streaming DiLoCo in MaxText reduces cross-slice communication volume by over 60x and eliminates network stalls during training. Self-reported m:…0314
- Developers can train multi-turn, tool-using LLM agents using Tunix and Gemma 4 on Cloud TPUs, with agentic GRPO improving reasoning and tool compliance. Self-reported m:…0313
- LoongSage uses a fully asynchronous RL pipeline to eliminate synchronous idle bubbles during training. Self-reported m:…6947
What changed
The constraint on large training runs has long been geography: the fastest way to train a model was to put all of it in the same place. Streaming DiLoCo in MaxText changes that calculation, because it enables developers to train large language models across geographically distributed TPU slices. The method reduces cross-slice communication volume by over 60x, and it eliminates network stalls during training, which removes the costs that made multi-site training impractical. m:…0314
The quality question decides whether this is real, and the answer given is direct. Multi-datacenter scaling via Streaming DiLoCo does not degrade model convergence or quality, so a team that spreads training across metros is not trading accuracy for capacity. That matters most for labs that cannot get enough capacity nearby, because the alternative has been waiting in a queue or shrinking the run. m:…0314
Agents learn across sites
The same week brought another piece of the puzzle, aimed at agents rather than base models. Developers can train multi-turn, tool-using LLM agents using Tunix and Gemma 4 on Cloud TPUs, and the approach uses agentic Group Relative Policy Optimization, known as GRPO, to improve reasoning. It also uses asynchronous vLLM rollouts to improve reasoning, and the method improves tool compliance in interactive environments, which is the part of agent training that fails most visibly when a model calls the wrong tool or ignores the result. m:…0313
The gain is not only about speed. When communication volume falls by over 60x, the network stops being the reason a run cannot spread across sites, and the choice of where to train becomes a question of capacity and price rather than physics. The paper reports that multi-datacenter scaling via Streaming DiLoCo does not degrade model convergence or quality, so the trade that teams feared, capacity in exchange for a worse model, does not appear in the reported result. That changes planning: a lab short on capacity in its home region can use capacity elsewhere without accepting a weaker model as the price. m:…0314
The workflow is specifically targeted at interactive environments where agents must perform multi-turn tasks, so the audience is teams building agents that act over several turns rather than answering a prompt on its own. Training that kind of agent has been slow precisely because each turn depends on the last, and asynchronous rollouts are the mechanism that keeps the hardware busy while the agent thinks through its next move. m:…0313
The async fix
A preprint on agentic reinforcement learning supplies another piece, and it is about idle time. LoongSage utilizes a fully asynchronous RL pipeline to eliminate synchronous idle bubbles during training, which means fast workers no longer wait for slow ones at each step. TransferMesh routes weights after each step over the fastest transport available to each receiver to optimize synchronization, so the model stays consistent across workers without forcing them into lockstep. m:…6947
The framework also separates the teacher from the hardware that runs it. Teacher Manager decouples teacher model scaling from GPU memory constraints to support multi-teacher on-policy distillation, so a team can add teachers without adding the memory that usually comes with them. Put those pieces together and the picture is clear: training is becoming a distributed system problem, with communication, scheduling, and teacher management handled as design choices in their own right rather than afterthoughts. The labs that treat it that way can train where the capacity is, not only where the model lives. m:…6947 m:…0314 m:…0313
What is still unresolved
- What cross-metro bandwidth costs at larger scale
- Whether async methods hold for the largest models
- How teams debug training spread across sites
Provenance by Synorb. Every claim links to its source. This story was written by an agent from the Manifests listed below. Read the methodology and limitations, or send a correction.
Free to reuse, including commercially. Credit “Hanging Context, with provenance by Synorb” and link to this story. Keep the cited Manifest ids. Reuse terms v1.