The GPU bottleneck moved from buying chips to starting them

Minutes you cannot bill

Initializing a large language model workload can take minutes, Nvidia says, and cold starts occur when inference demand spikes or a replica fails. The blunt consequence is that available GPUs do not translate immediately into serving capacity during cold starts. A team that buys enough GPUs for its peak still has to wait for those GPUs to become useful, and the wait happens at exactly the moment demand is highest: a traffic spike, or the moment a replica dies and its replacement has to load a model from scratch. The bottleneck has moved. The industry spent two years arguing about who could buy chips; the argument now is about how fast a chip you already own can start doing work. m:…3919

LinkedIn's answer, described in a preprint about its Firepool system, is to keep capacity warm. Firepool reduces p90 pod start latency from 59.1 seconds to 3.5 seconds, the paper reports, and achieves a 94.7% fulfillment rate for pod requests using warm or standby capacity. The mechanism is deliberately boring: building pods from a shared base image and binding job code at runtime makes pods interchangeable, which makes warm-pool management efficient. In plain terms, if every pod starts from the same image and only the job differs, you can keep a pool of ready pods waiting and hand them jobs as they arrive, instead of building each pod when the job shows up. The 94.7% fulfillment rate is the number that matters for an operator, because a warm pool that is empty when you need it is just a cold pool with extra steps. This is a preprint, so the numbers are the authors' own results, but the design is the point: the fix is interchangeability, not more hardware. m:…9731

Restore instead of start

Nvidia's answer is different: do not start the workload at all, restore it. NVIDIA Snapshot is a new open-source component of the NVIDIA DSX AI Factory Platform, and it makes initialized GPU workloads reusable across inference replicas. Snapshot reduces startup overhead by capturing fully initialized GPU workers, so a replica that would have spent minutes loading a model instead resumes from a saved state. Crusoe Cloud demonstrated a 15x speedup for GLM-5.3 startup using Snapshot, and pod creation to Ready time fell from 50 seconds to 31 seconds. The mechanism uses CRIU and NVIDIA cuda-checkpoint, and Snapshot is a Kubernetes-native checkpoint and restore system, which means it fits into the orchestration most teams already run rather than asking them to adopt a new one. m:…3919 m:…1490

The detail that decides whether restore beats warm pools is where the saved state lives. Restoring 594 GB of GPU state from local NVMe took 15 seconds in Crusoe's demonstration, while restoring the same state from NFS took 33.3 seconds. That gap is the whole argument in one number: restore is fast when the state is close, and the storage layer becomes the new bottleneck when it is not. Crusoe also contributed fixes upstream, including one for a restore failure caused by a missing mount namespace, which is the kind of unglamorous work that decides whether a feature survives contact with production. A checkpoint system that fails on a namespace edge case is a system operators will not trust with a replica that has to come back during a spike. m:…1490

The Kubernetes layer catches up

The third piece of the week is the orchestration layer itself. Nvidia announced two open-source projects for building and operating managed Kubernetes services on GPU infrastructure: NVIDIA Kubernetes Engine and NVIDIA Kubernetes Recipes, both currently available in public preview. AI Factory customers seek to consume GPU capacity using existing Kubernetes APIs and tools, Nvidia says, which is a polite way of saying that customers do not want a second control plane for their GPUs. The projects matter to the cold-start story because warm pools and snapshot restores both need somewhere to live: a pool of ready pods is a Kubernetes concept, and a restore is a pod lifecycle event. If the standard orchestrator can express warm capacity and fast restore as normal operations, the techniques stop being research projects and become configuration. The decision for a platform team is now a comparison, not a purchase: warm pools cost idle capacity, restores cost storage and complexity, and both beat buying more GPUs to cover a delay that hardware alone cannot fix. m:…3920