---
title: Serving is the new model race: Baseten and NVIDIA squeeze seconds out of weight syncs and storage
description: Model releases get headlines, but this week's infrastructure news is about the seconds between request and answer. Baseten cut weight-sync times and added scheduled pre-warming; NVIDIA published guidance on cutting storage out of the GPU's path. Latency is being bought back in the serving layer.
canonical: https://hangingcontext.com/stories/2026-10-31/inference-serving-squeezes-seconds/
last-updated: 2026-10-31
url: https://hangingcontext.com/stories/2026-10-31/inference-serving-squeezes-seconds/
published: 2026-10-31T00:35:52Z
edition: 2026-10-31T00
newsroom: artificial-intelligence
beat: infrastructure
trust: reported
source_count: 3
cited_manifest_ids: ['1793399446429912031', '1793380300749689878', '1793385786731655137']
---

# Serving is the new model race: Baseten and NVIDIA squeeze seconds out of weight syncs and storage

*Model releases get headlines, but this week's infrastructure news is about the seconds between request and answer. Baseten cut weight-sync times and added scheduled pre-warming; NVIDIA published guidance on cutting storage out of the GPU's path. Latency is being bought back in the serving layer.*

Reported · 3 sources · Artificial Intelligence · Published 2026-10-31 00:35 UTC

## Key facts

- Baseten says it cut sync time for 770GB of model weights from 36 seconds to 15 seconds and request pauses from 6 seconds to 0.8 seconds. (self-reported) [cites: 1793399446429912031]
- Baseten says KV cache offloading, enabled by removing the CPU checkpoint copy, improved output throughput per GPU by 3.2x at a concurrency of 48. (self-reported) [cites: 1793399446429912031]
- Baseten launched autoscaling schedules that raise minimum replicas before predictable traffic arrives, with up to 10 schedules per environment. (self-reported) [cites: 1793380300749689878]
- NVIDIA says GPUDirect Storage provides a direct data path between GPUs and storage targets to increase bandwidth and reduce CPU and memory utilization. (self-reported) [cites: 1793385786731655137]

## Fifteen seconds instead of thirty-six

Baseten says it rebuilt its delta weight synchronization path to use weights already on the GPU, eliminating the need for a second full copy in CPU memory. The headline numbers are for a 770GB checkpoint: sync time fell from 36 seconds to 15 seconds, and the request pause during synchronization fell from 6 seconds to 0.8 seconds. The delta decoder uses parallelism-aware logic to split deltas by the GPU that owns each change, update planning runs in a separate CUDA-free helper process to avoid contention with the SGLang scheduler, and delta checksum updates are computed in constant time from the position, old value, and new value of changed bytes. The Kimi K3 model sync on an 8.4 GB delta takes 75 seconds from request to live weights. [cites: 1793399446429912031]

The second-order effect matters more than the sync time itself. Freed host memory from removing the CPU checkpoint allows KV cache offloading, and Baseten says that offloading improved output throughput per GPU by 3.2 times at a concurrency of 48, and that Kimi K3 can now run on two GB300 nodes, a configuration previously blocked by CPU memory constraints. The company frames the improvement as enabling higher throughput for agentic workloads with long prefixes. All of these figures are the vendor's own measurements on its own platform, but the mechanism is specific enough to be checked: remove a copy, free memory, spend it on cache. [cites: 1793399446429912031]

## Pre-warm the replicas before the traffic

Baseten also launched autoscaling schedules, which let users proactively scale replicas before traffic arrives instead of reacting to it. Cold-start time includes allocating compute resources, starting the serving software, and loading the model, and schedules set a time window plus minimum and maximum replica counts for that window. Schedules can recur, for example weekday mornings, or run once, for example a product launch, and can be configured through the dashboard, CLI, or Management API. Each environment allows up to 10 schedules, all sharing one timezone, windows cannot overlap, and each window must last at least five minutes, with default settings applying when no schedule is active. [cites: 1793380300749689878]

> **From Synorb** — Claims with a source link. A Manifest holds a Signal of structured claims and a Brief, plus the source it came from. Your agent can check a sentence against it. [Read the guide](https://synorb.com/connect/start)


The idea is unglamorous and that is the point: for predictable traffic, the cheapest latency improvement is knowing when the traffic comes. Autoscaling schedules do not make any single request faster; they make sure a warm replica exists when the request arrives. For teams running agentic workloads, where a cold start can stall a multi-step task rather than a single response, that distinction is the product. [cites: 1793380300749689878]

## Take storage out of the GPU's path

NVIDIA's developer post on GPUDirect Storage describes the same squeeze applied to storage: GDS provides a direct data path between NVIDIA GPUs and storage targets, designed to increase bandwidth, minimize latency, and reduce CPU and memory utilization in on-premises and cloud environments, across a broad range of platforms, application frameworks, and workload patterns. The post is a configuration guide rather than a product launch, which is itself informative: the technology exists and the work now is checking and configuring it correctly on each platform. [cites: 1793385786731655137]

Taken together, the three announcements describe a serving stack being tuned at every layer: how weights move, how replicas warm, and how data reaches the GPU. None of them changes what a model can do. All of them change what it costs to run one at scale, and how long users wait. For teams choosing where to spend engineering time, the serving layer currently offers measurable, checkable wins, while the next model release offers a benchmark number. [cites: 1793399446429912031, 1793380300749689878, 1793385786731655137]

## What is still unresolved

- Baseten's throughput and sync figures are vendor measurements on its own platform, not independent benchmarks.
- Whether autoscaling schedules reduce costs as well as latency depends on how accurately traffic is predicted.
- The GPUDirect Storage post is configuration guidance, not a new product or measured result.

## Decision card

- **Useful for:** Teams running inference at scale; Anyone sizing GPU infrastructure for agentic workloads
- **Not for:** Anyone choosing between foundation models; Anyone expecting new model capabilities
- **Compare against:** Reactive autoscaling; CPU-mediated storage paths

## Provenance ledger

| Manifest id | Title | Source | Claims used | Published |
|---|---|---|---|---|
| `1793399446429912031` | The fastest delta weight syncs in the world | [baseten-blog](https://baseten.co/blog/the-fastest-delta-weight-syncs-in-the-world) | 1793399772233324683, 1793399772236770323, 1793399772240665899, 1793399772239833681, 1793399772237806940, 1793399772243630979, 1793399772244739379, 1793399772246952638, 1793399772247293249, 1793399772242562337 | 10-30 00:00 |
| `1793380300749689878` | Autoscaling schedules: reduce cold-start delays for predictable traffic | [baseten-blog](https://baseten.co/blog/autoscaling-schedules-reduce-cold-start-delays-for-predictable-traffic) | 1793380527057581526, 1793380527063912294, 1793380527062739059, 1793380527060886942, 1793380527065757074, 1793380527066278323, 1793380527068665332, 1793380527069473935, 1793380527072239252, 1793380527070784457 | 10-30 00:00 |
| `1793385786731655137` | How to Check and Configure NVIDIA GPUDirect Storage for Your Platform | [nvidia-developer-blog](https://developer.nvidia.com/blog/how-to-check-and-configure-nvidia-gpudirect-storage-for-your-platform) | 1793386692529200317, 1793386692530890798, 1793386692534408731, 1793386692536043164, 1793386692526096099, 1793386692532765631, 1793386692538114478 | 10-30 00:00 |

## Run these Manifests yourself

Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.

1. **Get a Synorb Key.** Click Generate Free Agent Key at https://synorb.com/connect/start, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{"client":"your-agent-name"}`. The key is the `api_key` field of the answer. Save the key. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons.
2. **Read each cited Manifest by id.** REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief.
3. **Check the claim.** Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them.

Cited Manifest ids:

- `1793399446429912031`
- `1793380300749689878`
- `1793385786731655137`

```sh
for id in 1793399446429912031 1793380300749689878 1793385786731655137; do
  curl -s -H "Authorization: Bearer $SYNORB_KEY" "https://api.synorb.com/manifests/by-id/$id"
done
```

> **From Synorb** — Ask what Synorb covers. The MCP tool synorb-stream-search answers questions like "do you have this publisher?" without using any Manifest allowance. [Connect and ask](https://synorb.com/connect/start)

## Related stories

- [The model you downloaded may be broken, weakened, or steered — and your checks will not catch it](https://hangingcontext.com/stories/2026-10-31/downloaded-model-trust-gap/)
- [The agent stack beyond the model: sandboxes, harnesses and memory](https://hangingcontext.com/stories/2026-10-30/agent-infrastructure-beyond-the-model/)
- [Coding agents are failing the test, not the task](https://hangingcontext.com/stories/2026-10-30/coding-agents-need-better-tests-and-specs/)
- [From prompt to playable: media agents learn to run the production](https://hangingcontext.com/stories/2026-10-30/agentic-media-from-prompt-to-production/)

## Links

- HTML: https://hangingcontext.com/stories/2026-10-31/inference-serving-squeezes-seconds/
- JSON: https://hangingcontext.com/stories/2026-10-31/inference-serving-squeezes-seconds/index.json
- Markdown: https://hangingcontext.com/stories/2026-10-31/inference-serving-squeezes-seconds/index.md

Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.

---
Reuse (terms v1): Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product. Credit: "Hanging Context, with provenance by Synorb", with a link to https://hangingcontext.com/stories/2026-10-31/inference-serving-squeezes-seconds/. Terms: https://hangingcontext.com/terms/reuse/