The serving stack is the new model race: sandboxes, tuned Nemotron, faster collectives

A cloud built for agents

Baseten has launched Baseten Sandboxes, the first component of a cloud purpose-built for agents. The sandboxes use microVMs to provide hardware-level isolation for untrusted code, allow agents to run long tasks securely and statefully with instant pause and resume, and can boot from standby in 25 milliseconds. They preserve filesystem, memory, and active processes when an agent disconnects, and allow compute to scale to zero while the environment sleeps. Sandboxes are free to run until November 16th. m:…2478

The roadmap shows what an agent cloud actually needs beyond a sandbox. Baseten plans to release Agent Drive, a shared, distributed, remote filesystem for sandboxes, and Agent Proxy to control agent connectivity and perform credential injection outside the sandbox so the agent never sees the credentials. The pattern is familiar from the earlier stories in this edition: the agent is the easy part, and the state, the files, and the secrets are the hard part. A sandbox that forgets everything when the agent disconnects is a demo; one that resumes is infrastructure. m:…2478

Baseten also joined the NVIDIA Nemotron Coalition to accelerate adoption of customized open-source models in production. Its Loops product connects managed reinforcement-learning workflows with deployment to dedicated inference, and it developed reinforcement-learning environments to teach models to follow complex instruction hierarchies. The coalition's bet is that the winning model for a given enterprise task will be a tuned open model, not a frontier rental — and the infrastructure to tune, serve, and measure it is the product. m:…9841

Tuning beats size

Surge AI's work is the evidence for that bet. It applied reinforcement learning to NVIDIA's Nemotron 3.5 Super VL, a 120-billion-parameter model with 12 billion active parameters, using four specialized datasets and Thinking Machines' Tinker platform. The trained checkpoints improved on all 12 evaluated benchmarks by 4.5 to 18.6 percentage points, and the model matched or exceeded scores from GPT-5.6 Terra on SWE-Bench Pro V2. Training eliminated roughly 40% of the base model's remaining failures on that benchmark. m:…6963

The more telling gains are in workflow completion, not benchmark scores. End-to-end workflow completion on AutomationBench increased from 15% to 28% after training, and the model improved on 51 of 65 tasks in HANDBOOK.md across ten simulated companies and five industries. The trained model increased its Slack history checks by 3 times and calendar checks by more than 2 times before taking action, and the share of workflows where it never opened the inbox or Slack fell from 75% to 40%. Success on requirements dependent on email or Slack climbed from under one-third to roughly 80%. The model got better at the unglamorous part of agentic work: checking the record before acting. m:…6963

Baseten's own customization numbers point the same way. CodeRabbit's agreement with its routing policy increased from 75.8% to 80.7% after customization, and CodeRabbit achieved 50.4% lower inference cost at saturated throughput after customizing Nemotron 3.5 Lightning for code review routing. A single NVIDIA Hopper GPU sustained up to 100 concurrent real-time audio streams over WebSocket using Nemotron 3.5 ASR Streaming. The pattern is consistent: a tuned model that fits the task beats a bigger model that does not, and the savings show up in cost per request, not in a leaderboard. m:…9841

The network is the bottleneck now

Astera Labs explains why serving is getting harder even as models get cheaper. Inference workloads are overtaking training workloads in volume, and AllReduce — the collective operation that synchronizes GPUs — accounts for 47% of prefill latency and 25% of decode latency in 8-GPU clusters, and an increasing percentage of inference time as cluster sizes grow. Peak server floating-point performance has scaled at about 3.0 times every two years, while interconnect bandwidth has grown by only about 1.4 times every two years. The compute got faster and the wires did not keep up. m:…3734

In-network compute attacks the problem by offloading collective operations directly to network switches, collapsing multi-step operations into a single pipelined process. Standard ring AllReduce with N participants requires N-1 communication steps per phase; in-network compute removes the repeated per-step overhead that dominates ring execution, and is most impactful at small message sizes, where a 2x speedup is possible with 8-GPU clusters. Small messages also enable finer-grained load balancing and improve tail latency. The implication for a team serving tuned models is that the serving bill is now a networking bill as much as a GPU bill, and the next round of optimization will happen in the switch. m:…3734