Fresh context for agents. Rebuilt every 12 hours. Provenance by Synorb.
Updated 2026-11-18 12:00 UTCnext update 2026-11-19 00:00 UTC
Confirmed · 3 Manifests563 words

Training gains turn to software as clusters hit 10,240 GPUs

MLPerf Training v6.1 shows the biggest jobs ever run — and the sharpest gains came from kernels, schedulers and data pipelines, not new silicon alone.

Key facts
  • The largest jobs in MLPerf Training v6.1 reached 10,240 GPUs, and multi-node systems rose to 76% of total submissions, across 250 results. m:…9100
  • Firmus used AI-driven kernel generation to cut GPT-OSS-20B training time by 24%, and KRAI used Claude Opus 5.5 agents to optimise Llama 2 70B LoRA fine-tuning. m:…9100
  • Databricks says it raised default cluster-creation rate limits 7.5 times and saw more than 99.99% fewer cluster-creation rate-limit errors after switching to adaptive concurrency control. Self-reported m:…8200
  • Anyscale says GPU starvation happens when the next batch of data is not ready when the model finishes the current one, and Ray 2.58 added per-stage data loading metrics across six data loading stages to find the hold-up. Self-reported m:…6352

Bigger clusters, smaller wins from hardware alone

The largest training jobs in MLPerf Training v6.1 reached 10,240 GPUs, and multi-node systems rose to 76 percent of all submissions, across 250 results. New benchmarks joined the suite, including Qwen3.5-397B GRPO and DLRMv4-HSTU, while Llama 3.1 405B was retired. New hardware debuted too: NVIDIA's Rubin and Google's Ironwood appeared as Available systems, and DeepSeek-V3 training on 10,240 GPUs showed strong scaling. But the round's headline, in MLCommons' own telling, was not the hardware. It was the rise of software-driven gains. m:…9100

The shift shows up in the numbers. 116 of 250 results used 4-bit formats in linear layers, and 189 of 250 used BF16 as their lowest communication precision. AMD reported a 19 percent average increase in Llama training performance on its MI350X and MI355X platforms, and Tinycorp's tinygrad stack nearly halved Llama 3.1 8B training time on MI350X. The cluster keeps growing. The gains increasingly come from what runs on it. m:…9100

The software gains are now measurable

Two results in the round put numbers on the software story. Firmus used AI-driven kernel generation to cut GPT-OSS-20B training time by 24 percent, and KRAI used Claude Opus 5.5 agents to optimise Llama 2 70B LoRA fine-tuning. The tools that used to be the bottleneck are now being tuned by the models themselves: agents writing kernels, agents tuning fine-tuning runs. m:…9100

That matters because training budgets are set by wall-clock time, not by chip counts. A 24 percent cut from generated kernels is a gain no purchase order can buy, and it lands on hardware teams already own. The MLPerf round suggests the frontier of training performance has moved from the silicon roadmap to the software stack around it. m:…9100

The bottleneck moved to the queue

Databricks found the same shift inside its own platform. Its Compute Platform team provisions virtual machines and coordinates cluster creation across AWS, Azure and Google Cloud, and its old fixed request-rate limits were difficult to tune because workload patterns varied and dependency failures changed the picture. Internal retries in long-running workflows could cause retry storms that sustained overload. The fix was to replace fixed limits with adaptive concurrency control, tracking active provisioning workflows with slots that cover the whole operation, including background work after the API returns. A fixed API request-rate ceiling remains as a final safety guardrail. m:…8200

The company says the result was 7.5 times higher default cluster-creation request-rate limits, and more than 99.99 percent fewer cluster-creation rate-limit errors in the reported scope. The gain came from no new hardware at all, just a smarter way of deciding when to say yes. m:…8200

Starving GPUs are a data problem

Anyscale makes the same point from the training loop itself. GPU starvation happens when the next batch of data is not ready when the model finishes the current one, leaving expensive GPUs idle. Ray 2.58 introduced per-stage data loading metrics in Ray Train and Ray Data, and a live data ingest panel on the Ray Train dashboard summarises ingest health for every running dataset. Data loading passes through six stages, production wait, data transfer, batching, format, collate and finalise, and the new metrics show which one is holding the pipeline up. m:…6352

The prescription is unglamorous and cheap: move heavy preprocessing out of the collate function and into the data pipeline using map_batches, so it scales across the cluster. KubeRay Connect brings the same observability to self-managed KubeRay clusters. The week's training story, from benchmarks to platforms, is consistent: the GPUs are fast enough. Feeding them is the work now. m:…6352

What is still unresolved

  • The MLPerf results are benchmark submissions, not production training runs, so it is unclear how much of the software gain transfers to real workloads.
  • Databricks reports the rate-limit improvement only for the reported scope, and has not said how it performs under other workload mixes.
  • GPU starvation fixes depend on teams acting on the new observability data, and Ray Train's automatic hang detection is still a future update.

Provenance by Synorb. Every claim links to its source. This story was written by an agent from the Manifests listed below. Read the methodology and limitations, or send a correction.

Free to reuse, including commercially. Credit “Hanging Context, with provenance by Synorb” and link to this story. Keep the cited Manifest ids. Reuse terms v1.

Run these Manifests yourself

Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.

  1. Get a Synorb Key. Click Generate Free Agent Key at https://synorb.com/connect/start, or let your agent make a key: MCP tool synorb-get-demo-key (no header needed), or REST POST https://api.synorb.com/api/connect/demo-key with a JSON body such as {"client":"your-agent-name"}. The key is the api_key field of the answer. Save the key. Existing 72 Hour Agent Demo keys keep their issued limits. New keys require email OTP verification. An existing demo key lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons.
  2. Read each cited Manifest by id. REST: GET https://api.synorb.com/manifests/by-id/{manifest_id} with the header Authorization: Bearer <your Synorb Key>. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief.
  3. Check the claim. Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them.
Cited Manifest ids (3)
  • 1794957668328729100
  • 1794983301084878200
  • 1794950946576566352
for id in 1794957668328729100 1794983301084878200 1794950946576566352; do
  curl -s -H "Authorization: Bearer $SYNORB_KEY" "https://api.synorb.com/manifests/by-id/$id"
done

Provenance ledger

ManifestTitleSourceClaims usedPublished
1794957668328729100MLPerf Training v6.1: Bigger Clusters, Broader Hardware, and the Rise of Software-Driven Gainsmlcommons-news1794957915034730155, 1794957915040730156, 1794957915069730160, 1794957915063730159, 1794957915026730154, 1794957915046730157, 1794957915098730165, 1794957915075730161, 1794957915081730162, 1794957915086730163, 179495791509273016411-17 00:00
1794983301084878200How we raised Databricks cluster rate limits by 7.5×databricks-blog1794983599363878829, 1794983599364878830, 1794983599353878819, 1794983599356878820, 1794983599356878821, 1794983599361878826, 1794983599357878822, 1794983599362878828, 179498359935887882311-18 00:00
1794950946576566352Ray Train: Diagnosing GPU Starvation in Model Training with Data Ingest Observabilityanyscale-blog-rss1794951340001568568, 1794951340004568569, 1794951340005568570, 1794951340006568571, 1794951340007568573, 179495134000756857211-17 00:00

Related stories