---
title: Training gains turn to software as clusters hit 10,240 GPUs
description: MLPerf Training v6.1 shows the biggest jobs ever run — and the sharpest gains came from kernels, schedulers and data pipelines, not new silicon alone.
canonical: https://hangingcontext.com/stories/2026-11-18/training-gains-turn-to-software/
last-updated: 2026-11-18
url: https://hangingcontext.com/stories/2026-11-18/training-gains-turn-to-software/
published: 2026-11-18T12:41:53Z
edition: 2026-11-18T12
newsroom: artificial-intelligence
beat: infrastructure
trust: confirmed
source_count: 3
cited_manifest_ids: ['1794957668328729100', '1794983301084878200', '1794950946576566352']
---

# Training gains turn to software as clusters hit 10,240 GPUs

*MLPerf Training v6.1 shows the biggest jobs ever run — and the sharpest gains came from kernels, schedulers and data pipelines, not new silicon alone.*

Confirmed · 3 Manifests · Artificial Intelligence · Published 2026-11-18 12:41 UTC

## Key facts

- The largest jobs in MLPerf Training v6.1 reached 10,240 GPUs, and multi-node systems rose to 76% of total submissions, across 250 results. [cites: 1794957668328729100]
- Firmus used AI-driven kernel generation to cut GPT-OSS-20B training time by 24%, and KRAI used Claude Opus 5.5 agents to optimise Llama 2 70B LoRA fine-tuning. [cites: 1794957668328729100]
- Databricks says it raised default cluster-creation rate limits 7.5 times and saw more than 99.99% fewer cluster-creation rate-limit errors after switching to adaptive concurrency control. (self-reported) [cites: 1794983301084878200]
- Anyscale says GPU starvation happens when the next batch of data is not ready when the model finishes the current one, and Ray 2.58 added per-stage data loading metrics across six data loading stages to find the hold-up. (self-reported) [cites: 1794950946576566352]

## Bigger clusters, smaller wins from hardware alone

The largest training jobs in MLPerf Training v6.1 reached 10,240 GPUs, and multi-node systems rose to 76 percent of all submissions, across 250 results. New benchmarks joined the suite, including Qwen3.5-397B GRPO and DLRMv4-HSTU, while Llama 3.1 405B was retired. New hardware debuted too: NVIDIA's Rubin and Google's Ironwood appeared as Available systems, and DeepSeek-V3 training on 10,240 GPUs showed strong scaling. But the round's headline, in MLCommons' own telling, was not the hardware. It was the rise of software-driven gains. [cites: 1794957668328729100]

The shift shows up in the numbers. 116 of 250 results used 4-bit formats in linear layers, and 189 of 250 used BF16 as their lowest communication precision. AMD reported a 19 percent average increase in Llama training performance on its MI350X and MI355X platforms, and Tinycorp's tinygrad stack nearly halved Llama 3.1 8B training time on MI350X. The cluster keeps growing. The gains increasingly come from what runs on it. [cites: 1794957668328729100]

## The software gains are now measurable

Two results in the round put numbers on the software story. Firmus used AI-driven kernel generation to cut GPT-OSS-20B training time by 24 percent, and KRAI used Claude Opus 5.5 agents to optimise Llama 2 70B LoRA fine-tuning. The tools that used to be the bottleneck are now being tuned by the models themselves: agents writing kernels, agents tuning fine-tuning runs. [cites: 1794957668328729100]

> **From Synorb** — Connect your agent over MCP. Add https://mcp.synorb.com/mcp to your agent, then call the tool synorb-get-demo-key if it has no key yet. It needs no email and no form. Keys are free for 72 hours, and new keys are limited per network. [How to connect](https://synorb.com/connect/start)


That matters because training budgets are set by wall-clock time, not by chip counts. A 24 percent cut from generated kernels is a gain no purchase order can buy, and it lands on hardware teams already own. The MLPerf round suggests the frontier of training performance has moved from the silicon roadmap to the software stack around it. [cites: 1794957668328729100]

## The bottleneck moved to the queue

Databricks found the same shift inside its own platform. Its Compute Platform team provisions virtual machines and coordinates cluster creation across AWS, Azure and Google Cloud, and its old fixed request-rate limits were difficult to tune because workload patterns varied and dependency failures changed the picture. Internal retries in long-running workflows could cause retry storms that sustained overload. The fix was to replace fixed limits with adaptive concurrency control, tracking active provisioning workflows with slots that cover the whole operation, including background work after the API returns. A fixed API request-rate ceiling remains as a final safety guardrail. [cites: 1794983301084878200]

The company says the result was 7.5 times higher default cluster-creation request-rate limits, and more than 99.99 percent fewer cluster-creation rate-limit errors in the reported scope. The gain came from no new hardware at all, just a smarter way of deciding when to say yes. [cites: 1794983301084878200]

## Starving GPUs are a data problem

Anyscale makes the same point from the training loop itself. GPU starvation happens when the next batch of data is not ready when the model finishes the current one, leaving expensive GPUs idle. Ray 2.58 introduced per-stage data loading metrics in Ray Train and Ray Data, and a live data ingest panel on the Ray Train dashboard summarises ingest health for every running dataset. Data loading passes through six stages, production wait, data transfer, batching, format, collate and finalise, and the new metrics show which one is holding the pipeline up. [cites: 1794950946576566352]

The prescription is unglamorous and cheap: move heavy preprocessing out of the collate function and into the data pipeline using map_batches, so it scales across the cluster. KubeRay Connect brings the same observability to self-managed KubeRay clusters. The week's training story, from benchmarks to platforms, is consistent: the GPUs are fast enough. Feeding them is the work now. [cites: 1794950946576566352]

## What is still unresolved

- The MLPerf results are benchmark submissions, not production training runs, so it is unclear how much of the software gain transfers to real workloads.
- Databricks reports the rate-limit improvement only for the reported scope, and has not said how it performs under other workload mixes.
- GPU starvation fixes depend on teams acting on the new observability data, and Ray Train's automatic hang detection is still a future update.

## Decision card

- **Useful for:** Executives setting AI training budgets and infrastructure plans; Platform teams deciding where the next training efficiency gains come from
- **Not for:** Teams expecting the next hardware generation alone to cut training times; Anyone planning training capacity on GPU counts without measuring data ingest
- **Compare against:** Databricks' adaptive concurrency work for platform-level software gains; Anyscale's Ray Train observability for finding data-pipeline bottlenecks

## Provenance ledger

| Manifest id | Title | Source | Claims used | Published |
|---|---|---|---|---|
| `1794957668328729100` | MLPerf Training v6.1: Bigger Clusters, Broader Hardware, and the Rise of Software-Driven Gains | [mlcommons-news](https://mlcommons.org/2026/11/mlperf-training-v6-1-insights) | 1794957915034730155, 1794957915040730156, 1794957915069730160, 1794957915063730159, 1794957915026730154, 1794957915046730157, 1794957915098730165, 1794957915075730161, 1794957915081730162, 1794957915086730163, 1794957915092730164 | 11-17 00:00 |
| `1794983301084878200` | How we raised Databricks cluster rate limits by 7.5× | [databricks-blog](https://databricks.com/blog/how-we-raised-databricks-cluster-rate-limits-75) | 1794983599363878829, 1794983599364878830, 1794983599353878819, 1794983599356878820, 1794983599356878821, 1794983599361878826, 1794983599357878822, 1794983599362878828, 1794983599358878823 | 11-18 00:00 |
| `1794950946576566352` | Ray Train: Diagnosing GPU Starvation in Model Training with Data Ingest Observability | [anyscale-blog-rss](https://anyscale.com/blog/diagnosing-gpu-starvation-with-ray-train) | 1794951340001568568, 1794951340004568569, 1794951340005568570, 1794951340006568571, 1794951340007568573, 1794951340007568572 | 11-17 00:00 |

## Run these Manifests yourself

Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.

1. **Get a Synorb Key.** Click Generate Free Agent Key at https://synorb.com/connect/start, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{"client":"your-agent-name"}`. The key is the `api_key` field of the answer. Save the key. Existing 72 Hour Agent Demo keys keep their issued limits. New keys require email OTP verification. An existing demo key lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons.
2. **Read each cited Manifest by id.** REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief.
3. **Check the claim.** Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them.

Cited Manifest ids:

- `1794957668328729100`
- `1794983301084878200`
- `1794950946576566352`

```sh
for id in 1794957668328729100 1794983301084878200 1794950946576566352; do
  curl -s -H "Authorization: Bearer $SYNORB_KEY" "https://api.synorb.com/manifests/by-id/$id"
done
```

> **From Synorb** — Save what you follow as a Beacon. A Beacon is a saved query that your agent pulls with an explicit lookback window. Every new account starts with 5 starter Beacons: Business & Technology Podcasts, AI Companies, Venture Capital, Private Companies and Public Company Filings & Executive Transitions. [See Beacons](https://synorb.com/connect/start)

## Related stories

- [Agents move into the back office: buying, shipping, and the data underneath](https://hangingcontext.com/stories/2026-11-22/agents-move-into-the-back-office/)
- [When agents do the work, proving who did it becomes the product](https://hangingcontext.com/stories/2026-11-22/proving-who-did-the-work/)
- [AI spend is now big enough to need its own budget discipline](https://hangingcontext.com/stories/2026-11-22/ai-spend-gets-a-budget/)
- [The attacker is now an agent — and the frameworks defenders use, turned around](https://hangingcontext.com/stories/2026-11-22/the-attacker-is-now-an-agent/)

## Links

- HTML: https://hangingcontext.com/stories/2026-11-18/training-gains-turn-to-software/
- JSON: https://hangingcontext.com/stories/2026-11-18/training-gains-turn-to-software/index.json
- Markdown: https://hangingcontext.com/stories/2026-11-18/training-gains-turn-to-software/index.md

Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.

---
Reuse (terms v1): Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product. Credit: "Hanging Context, with provenance by Synorb", with a link to https://hangingcontext.com/stories/2026-11-18/training-gains-turn-to-software/. Terms: https://hangingcontext.com/terms/reuse/
