Reinforcement learning is becoming rentable infrastructure

Rent the loop

Baseten has made Serverless Loops generally available. Loops is a training SDK for supervised fine-tuning and reinforcement learning on frontier open models, and it supports a 256k context length for both supervised fine-tuning and reinforcement learning. It uses a per-token billing model and a Tinker-compatible API, so teams write training code against a familiar interface while Baseten manages the GPUs, kernels and distributed training underneath. Trained models can deploy directly to Baseten Dedicated Inference, or users can download the trained weights in the Hugging Face Safetensors format and take them elsewhere. m:…2167

The pitch is that training ends where serving begins: Baseten says its production inference platform serves more than 15 billion inference requests per day, the company's own figure. Early adopters show the range of uses: NVIDIA improved intent classification accuracy from 76.8 percent to 94 percent using a post-trained Nemotron Super 3.5 VL model, Datadog uses Loops to test training ideas for its SRE agent, LangChain Labs uses Loops to post-train models on agent traces for the LangSmith Engine, and Ambience Healthcare uses Loops to handle clinical language nuances. Baseten plans to release new features including quantization techniques and score centering. m:…2167

The weight refresh problem

Renting the loop does not remove the plumbing underneath it. During reinforcement learning post-training, training trajectories are generated on the fly by running inference with the current policy model. For training efficiency, it is common to run inference and training in parallel on separate GPU resources in a disaggregated setup. That split creates a sync problem: the system must periodically refresh the weights on the training GPUs during reinforcement learning post-training, so training keeps learning from the model that is actually generating the trajectories. m:…5421

NVIDIA's answer, described in its developer blog, is the Reshard Collective, a mechanism to handle that periodic weight refresh in disaggregated training setups. The blog gives the shape of the idea rather than a full implementation. This suggests the refresh step is becoming a first-class piece of RL infrastructure, not a detail teams improvise: when inference and training live on different GPUs, keeping their weights in step is part of the training loop itself. m:…5421

Learning from live traffic

A separate piece of the same puzzle is what the model learns from. Production RL creates a continuous feedback loop where models learn from real-world experiences and improve, and production data from language model deployments contains experiences from completing economically valuable work. Instead of training only in synthetic environments, the model learns from the traffic it already serves. That is the attraction, and also the risk, because production traces are messier than curated training data. m:…9986

The recipe from Applied Compute is blunt about the failure modes. Retokenization of production traces causes production RL runs to collapse, because retokenizing concatenated text can change token boundaries and lead to training instability. Token IDs must therefore be gathered before the chat processor is applied, to preserve training integrity. Requesting logprobs from an inference deployment may slow it down by approximately 5 percent, which is a real cost of instrumenting live serving for training. m:…9986

What keeps production RL stable

Stability is where the recipe earns its keep. Score centering outperforms importance sampling in production RL, hillclimbing faster and higher, according to the authors. Asynchronous production RL can improve data efficiency by overlapping data collection and training, though it increases staleness, so faster redeployment cycles trade against fresher data. Sampled KL regularization significantly reduces run variance and increases the stability of the plateau in production RL. Taken together, these are practical rules for a loop that never stops: center the scores, regularize the updates, and accept that asynchronous training means learning from slightly older data. m:…9986

The environment side of multi-agent RL has its own bottleneck, and a preprint named SEER takes it on. SEER reduces environment-execution latency in multi-agent reinforcement learning workloads by modeling steps as bounded streams of timestamped events. It implements a CAS-based dual-layer event queue to manage per-entity causal order and expose parallel work, and its STPA mechanism improves thread-pool selection by integrating runtime traces with static workload and hardware features. In a preprint, SEER reports a 16.2 percent latency reduction in UAV swarm control and up to 64.9 percent in wireless management compared with existing simulators. Because this is a preprint reporting its authors' own results, those figures should be treated as claims to be reproduced, not settled measurements. m:…9599

What this changes for builders

Put the four pieces together and the direction is clear. Training is rentable, the weight refresh that keeps disaggregated training in sync is being productized, live traffic is becoming training data, and multi-agent environments are being rebuilt for parallel execution. A small team can now rent the loop, train on its own traces using the production RL recipe, and run multi-agent environments on a runtime like SEER. The cautions travel with the gains: Baseten's request volume and customer results are the vendor's own figures, the preprint numbers are the authors' own, and the production RL recipe warns plainly that a small data-handling mistake can collapse a run. Specific pricing details for the per-token billing model are not provided in the Baseten account, so the cost of renting the loop is still open. m:…2167 m:…9599 m:…9986