Serving is the new model race: Baseten and NVIDIA squeeze seconds out of weight syncs and storage

Fifteen seconds instead of thirty-six

Baseten says it rebuilt its delta weight synchronization path to use weights already on the GPU, eliminating the need for a second full copy in CPU memory. The headline numbers are for a 770GB checkpoint: sync time fell from 36 seconds to 15 seconds, and the request pause during synchronization fell from 6 seconds to 0.8 seconds. The delta decoder uses parallelism-aware logic to split deltas by the GPU that owns each change, update planning runs in a separate CUDA-free helper process to avoid contention with the SGLang scheduler, and delta checksum updates are computed in constant time from the position, old value, and new value of changed bytes. The Kimi K3 model sync on an 8.4 GB delta takes 75 seconds from request to live weights. m:…2031

The second-order effect matters more than the sync time itself. Freed host memory from removing the CPU checkpoint allows KV cache offloading, and Baseten says that offloading improved output throughput per GPU by 3.2 times at a concurrency of 48, and that Kimi K3 can now run on two GB300 nodes, a configuration previously blocked by CPU memory constraints. The company frames the improvement as enabling higher throughput for agentic workloads with long prefixes. All of these figures are the vendor's own measurements on its own platform, but the mechanism is specific enough to be checked: remove a copy, free memory, spend it on cache. m:…2031

Pre-warm the replicas before the traffic

Baseten also launched autoscaling schedules, which let users proactively scale replicas before traffic arrives instead of reacting to it. Cold-start time includes allocating compute resources, starting the serving software, and loading the model, and schedules set a time window plus minimum and maximum replica counts for that window. Schedules can recur, for example weekday mornings, or run once, for example a product launch, and can be configured through the dashboard, CLI, or Management API. Each environment allows up to 10 schedules, all sharing one timezone, windows cannot overlap, and each window must last at least five minutes, with default settings applying when no schedule is active. m:…9878

The idea is unglamorous and that is the point: for predictable traffic, the cheapest latency improvement is knowing when the traffic comes. Autoscaling schedules do not make any single request faster; they make sure a warm replica exists when the request arrives. For teams running agentic workloads, where a cold start can stall a multi-step task rather than a single response, that distinction is the product. m:…9878

Take storage out of the GPU's path

NVIDIA's developer post on GPUDirect Storage describes the same squeeze applied to storage: GDS provides a direct data path between NVIDIA GPUs and storage targets, designed to increase bandwidth, minimize latency, and reduce CPU and memory utilization in on-premises and cloud environments, across a broad range of platforms, application frameworks, and workload patterns. The post is a configuration guide rather than a product launch, which is itself informative: the technology exists and the work now is checking and configuring it correctly on each platform. m:…5137

Taken together, the three announcements describe a serving stack being tuned at every layer: how weights move, how replicas warm, and how data reaches the GPU. None of them changes what a model can do. All of them change what it costs to run one at scale, and how long users wait. For teams choosing where to spend engineering time, the serving layer currently offers measurable, checkable wins, while the next model release offers a benchmark number. m:…2031 m:…9878 m:…5137