Training gains turn to software as clusters hit 10,240 GPUs
MLPerf Training v6.1 shows the biggest jobs ever run — and the sharpest gains came from kernels, schedulers and data pipelines, not new silicon alone.
- The largest jobs in MLPerf Training v6.1 reached 10,240 GPUs, and multi-node systems rose to 76% of total submissions, across 250 results. m:…9100
- Firmus used AI-driven kernel generation to cut GPT-OSS-20B training time by 24%, and KRAI used Claude Opus 5.5 agents to optimise Llama 2 70B LoRA fine-tuning. m:…9100
- Databricks says it raised default cluster-creation rate limits 7.5 times and saw more than 99.99% fewer cluster-creation rate-limit errors after switching to adaptive concurrency control. Self-reported m:…8200
- Anyscale says GPU starvation happens when the next batch of data is not ready when the model finishes the current one, and Ray 2.58 added per-stage data loading metrics across six data loading stages to find the hold-up. Self-reported m:…6352
Bigger clusters, smaller wins from hardware alone
The largest training jobs in MLPerf Training v6.1 reached 10,240 GPUs, and multi-node systems rose to 76 percent of all submissions, across 250 results. New benchmarks joined the suite, including Qwen3.5-397B GRPO and DLRMv4-HSTU, while Llama 3.1 405B was retired. New hardware debuted too: NVIDIA's Rubin and Google's Ironwood appeared as Available systems, and DeepSeek-V3 training on 10,240 GPUs showed strong scaling. But the round's headline, in MLCommons' own telling, was not the hardware. It was the rise of software-driven gains. m:…9100
The shift shows up in the numbers. 116 of 250 results used 4-bit formats in linear layers, and 189 of 250 used BF16 as their lowest communication precision. AMD reported a 19 percent average increase in Llama training performance on its MI350X and MI355X platforms, and Tinycorp's tinygrad stack nearly halved Llama 3.1 8B training time on MI350X. The cluster keeps growing. The gains increasingly come from what runs on it. m:…9100
The software gains are now measurable
Two results in the round put numbers on the software story. Firmus used AI-driven kernel generation to cut GPT-OSS-20B training time by 24 percent, and KRAI used Claude Opus 5.5 agents to optimise Llama 2 70B LoRA fine-tuning. The tools that used to be the bottleneck are now being tuned by the models themselves: agents writing kernels, agents tuning fine-tuning runs. m:…9100
That matters because training budgets are set by wall-clock time, not by chip counts. A 24 percent cut from generated kernels is a gain no purchase order can buy, and it lands on hardware teams already own. The MLPerf round suggests the frontier of training performance has moved from the silicon roadmap to the software stack around it. m:…9100
The bottleneck moved to the queue
Databricks found the same shift inside its own platform. Its Compute Platform team provisions virtual machines and coordinates cluster creation across AWS, Azure and Google Cloud, and its old fixed request-rate limits were difficult to tune because workload patterns varied and dependency failures changed the picture. Internal retries in long-running workflows could cause retry storms that sustained overload. The fix was to replace fixed limits with adaptive concurrency control, tracking active provisioning workflows with slots that cover the whole operation, including background work after the API returns. A fixed API request-rate ceiling remains as a final safety guardrail. m:…8200
The company says the result was 7.5 times higher default cluster-creation request-rate limits, and more than 99.99 percent fewer cluster-creation rate-limit errors in the reported scope. The gain came from no new hardware at all, just a smarter way of deciding when to say yes. m:…8200
Starving GPUs are a data problem
Anyscale makes the same point from the training loop itself. GPU starvation happens when the next batch of data is not ready when the model finishes the current one, leaving expensive GPUs idle. Ray 2.58 introduced per-stage data loading metrics in Ray Train and Ray Data, and a live data ingest panel on the Ray Train dashboard summarises ingest health for every running dataset. Data loading passes through six stages, production wait, data transfer, batching, format, collate and finalise, and the new metrics show which one is holding the pipeline up. m:…6352
The prescription is unglamorous and cheap: move heavy preprocessing out of the collate function and into the data pipeline using map_batches, so it scales across the cluster. KubeRay Connect brings the same observability to self-managed KubeRay clusters. The week's training story, from benchmarks to platforms, is consistent: the GPUs are fast enough. Feeding them is the work now. m:…6352
What is still unresolved
- The MLPerf results are benchmark submissions, not production training runs, so it is unclear how much of the software gain transfers to real workloads.
- Databricks reports the rate-limit improvement only for the reported scope, and has not said how it performs under other workload mixes.
- GPU starvation fixes depend on teams acting on the new observability data, and Ray Train's automatic hang detection is still a future update.
Provenance by Synorb. Every claim links to its source. This story was written by an agent from the Manifests listed below. Read the methodology and limitations, or send a correction.
Free to reuse, including commercially. Credit “Hanging Context, with provenance by Synorb” and link to this story. Keep the cited Manifest ids. Reuse terms v1.