{
  "schema": "hangingcontext.desk/1",
  "type": "story",
  "id": "1244",
  "slug": "rl-training-for-everyone",
  "edition": "2026-11-07T00",
  "url": "https://hangingcontext.com/stories/2026-11-07/rl-training-for-everyone/",
  "newsroom": "artificial-intelligence",
  "beat": "infrastructure",
  "newsroom_title": "Artificial Intelligence",
  "angle": "Frontier reinforcement-learning training becomes rentable infrastructure",
  "headline": "Reinforcement learning is becoming rentable infrastructure",
  "dek": "Baseten now rents out the full RL training loop, NVIDIA is speeding up the weight refresh that keeps training in sync, and a production RL recipe plus a preprint runtime are tackling the two hard parts: learning from live traffic and running many agents at once.",
  "published_at": "2026-11-07T00:44:28Z",
  "labels": {
    "trust": "confirmed",
    "trust_text": "Confirmed · 4 Manifests",
    "newsroom": "Artificial Intelligence"
  },
  "source_count": 4,
  "publishers": [
    "applied-compute-research",
    "arxiv-ai-infra-inference-ops",
    "baseten-blog",
    "nvidia-developer-blog"
  ],
  "sources": [
    "applied-compute-research",
    "arxiv-ai-infra-inference-ops",
    "baseten-blog",
    "nvidia-developer-blog"
  ],
  "distinct_source_count": 4,
  "key_facts": [
    {
      "text": "Baseten has made Serverless Loops generally available, a training SDK for supervised fine-tuning and reinforcement learning on frontier open models, with a 256k context length for both.",
      "cites": [
        {
          "claim_ids": [
            "1793997954856453231",
            "1793997954858453232",
            "1793997954860453233"
          ],
          "manifest_id": "1793997728904452167"
        }
      ],
      "self_reported": false
    },
    {
      "text": "Baseten says its production inference platform serves more than 15 billion inference requests per day, and trained models can deploy directly to it or be exported in Hugging Face Safetensors format.",
      "cites": [
        {
          "claim_ids": [
            "1793997954861453234",
            "1793997954862453235",
            "1793997954866453237"
          ],
          "manifest_id": "1793997728904452167"
        }
      ],
      "self_reported": true
    },
    {
      "text": "NVIDIA improved intent classification accuracy from 76.8% to 94% using a post-trained Nemotron Super 3.5 VL model, according to Baseten.",
      "cites": [
        {
          "claim_ids": [
            "1793997954862453235"
          ],
          "manifest_id": "1793997728904452167"
        }
      ],
      "self_reported": true
    },
    {
      "text": "In production RL, retokenization of production traces causes runs to collapse, so token IDs must be gathered before the chat processor is applied.",
      "cites": [
        {
          "claim_ids": [
            "1794001130838471742",
            "1794001130837471741"
          ],
          "manifest_id": "1794000893011469986"
        }
      ],
      "self_reported": false
    },
    {
      "text": "In a preprint, SEER reports a 16.2% latency reduction in UAV swarm control and up to 64.9% in wireless management compared with existing simulators.",
      "cites": [
        {
          "claim_ids": [
            "1793953815086210709"
          ],
          "manifest_id": "1793953558428209599"
        }
      ],
      "self_reported": true
    }
  ],
  "images": [],
  "sections": [
    {
      "heading": "Rent the loop",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1793997954856453231",
                "1793997954858453232",
                "1793997954860453233",
                "1793997954875453243",
                "1793997954873453241",
                "1793997954862453235",
                "1793997954866453237"
              ],
              "manifest_id": "1793997728904452167"
            }
          ],
          "markdown": "Baseten has made Serverless Loops generally available. Loops is a training SDK for supervised fine-tuning and reinforcement learning on frontier open models, and it supports a 256k context length for both supervised fine-tuning and reinforcement learning. It uses a per-token billing model and a Tinker-compatible API, so teams write training code against a familiar interface while Baseten manages the GPUs, kernels and distributed training underneath. Trained models can deploy directly to Baseten Dedicated Inference, or users can download the trained weights in the Hugging Face Safetensors format and take them elsewhere."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1793997954861453234",
                "1793997954862453235",
                "1793997954868453238",
                "1793997954869453239",
                "1793997954871453240",
                "1793997954873453241"
              ],
              "manifest_id": "1793997728904452167"
            }
          ],
          "markdown": "The pitch is that training ends where serving begins: Baseten says its production inference platform serves more than 15 billion inference requests per day, the company's own figure. Early adopters show the range of uses: NVIDIA improved intent classification accuracy from 76.8 percent to 94 percent using a post-trained Nemotron Super 3.5 VL model, Datadog uses Loops to test training ideas for its SRE agent, LangChain Labs uses Loops to post-train models on agent traces for the LangSmith Engine, and Ambience Healthcare uses Loops to handle clinical language nuances. Baseten plans to release new features including quantization techniques and score centering."
        }
      ]
    },
    {
      "heading": "The weight refresh problem",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1794005996212495601",
                "1794005996216495602",
                "1794005996218495603"
              ],
              "manifest_id": "1794005765998495421"
            }
          ],
          "markdown": "Renting the loop does not remove the plumbing underneath it. During reinforcement learning post-training, training trajectories are generated on the fly by running inference with the current policy model. For training efficiency, it is common to run inference and training in parallel on separate GPU resources in a disaggregated setup. That split creates a sync problem: the system must periodically refresh the weights on the training GPUs during reinforcement learning post-training, so training keeps learning from the model that is actually generating the trajectories."
        },
        {
          "cites": [
            {
              "claim_ids": [],
              "manifest_id": "1794005765998495421"
            }
          ],
          "markdown": "NVIDIA's answer, described in its developer blog, is the Reshard Collective, a mechanism to handle that periodic weight refresh in disaggregated training setups. The blog gives the shape of the idea rather than a full implementation. This suggests the refresh step is becoming a first-class piece of RL infrastructure, not a detail teams improvise: when inference and training live on different GPUs, keeping their weights in step is part of the training loop itself."
        }
      ]
    },
    {
      "heading": "Learning from live traffic",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1794001130835471740",
                "1794001130830471738",
                "1794001130833471739"
              ],
              "manifest_id": "1794000893011469986"
            }
          ],
          "markdown": "A separate piece of the same puzzle is what the model learns from. Production RL creates a continuous feedback loop where models learn from real-world experiences and improve, and production data from language model deployments contains experiences from completing economically valuable work. Instead of training only in synthetic environments, the model learns from the traffic it already serves. That is the attraction, and also the risk, because production traces are messier than curated training data."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1794001130838471742",
                "1794001130835471740",
                "1794001130837471741",
                "1794001130840471743"
              ],
              "manifest_id": "1794000893011469986"
            }
          ],
          "markdown": "The recipe from Applied Compute is blunt about the failure modes. Retokenization of production traces causes production RL runs to collapse, because retokenizing concatenated text can change token boundaries and lead to training instability. Token IDs must therefore be gathered before the chat processor is applied, to preserve training integrity. Requesting logprobs from an inference deployment may slow it down by approximately 5 percent, which is a real cost of instrumenting live serving for training."
        }
      ]
    },
    {
      "heading": "What keeps production RL stable",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1794001130841471744",
                "1794001130843471745",
                "1794001130844471746"
              ],
              "manifest_id": "1794000893011469986"
            }
          ],
          "markdown": "Stability is where the recipe earns its keep. Score centering outperforms importance sampling in production RL, hillclimbing faster and higher, according to the authors. Asynchronous production RL can improve data efficiency by overlapping data collection and training, though it increases staleness, so faster redeployment cycles trade against fresher data. Sampled KL regularization significantly reduces run variance and increases the stability of the plateau in production RL. Taken together, these are practical rules for a loop that never stops: center the scores, regularize the updates, and accept that asynchronous training means learning from slightly older data."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1793953815082210705",
                "1793953815084210707",
                "1793953815086210709"
              ],
              "manifest_id": "1793953558428209599"
            }
          ],
          "markdown": "The environment side of multi-agent RL has its own bottleneck, and a preprint named SEER takes it on. SEER reduces environment-execution latency in multi-agent reinforcement learning workloads by modeling steps as bounded streams of timestamped events. It implements a CAS-based dual-layer event queue to manage per-entity causal order and expose parallel work, and its STPA mechanism improves thread-pool selection by integrating runtime traces with static workload and hardware features. In a preprint, SEER reports a 16.2 percent latency reduction in UAV swarm control and up to 64.9 percent in wireless management compared with existing simulators. Because this is a preprint reporting its authors' own results, those figures should be treated as claims to be reproduced, not settled measurements."
        }
      ]
    },
    {
      "heading": "What this changes for builders",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1793997954861453234"
              ],
              "manifest_id": "1793997728904452167"
            },
            {
              "claim_ids": [
                "1793953815086210709"
              ],
              "manifest_id": "1793953558428209599"
            },
            {
              "claim_ids": [
                "1794001130838471742"
              ],
              "manifest_id": "1794000893011469986"
            }
          ],
          "markdown": "Put the four pieces together and the direction is clear. Training is rentable, the weight refresh that keeps disaggregated training in sync is being productized, live traffic is becoming training data, and multi-agent environments are being rebuilt for parallel execution. A small team can now rent the loop, train on its own traces using the production RL recipe, and run multi-agent environments on a runtime like SEER. The cautions travel with the gains: Baseten's request volume and customer results are the vendor's own figures, the preprint numbers are the authors' own, and the production RL recipe warns plainly that a small data-handling mistake can collapse a run. Specific pricing details for the per-token billing model are not provided in the Baseten account, so the cost of renting the loop is still open."
        }
      ]
    }
  ],
  "unresolved": [
    "The specific technical implementation of NVIDIA's Reshard Collective.",
    "Specific pricing details for Baseten's per-token billing model.",
    "Whether SEER's preprint latency reductions reproduce outside the authors' own tests.",
    "Mitigation strategies for the increased run variance observed with score centering in production RL."
  ],
  "decision_card": {
    "not_for": [
      "Anyone budgeting a training run from the per-token price, which Baseten has not disclosed",
      "Anyone treating preprint latency figures as settled benchmarks"
    ],
    "useful_for": [
      "Developers weighing rented RL training against running their own training stack",
      "Teams planning to train from production traffic rather than synthetic data"
    ],
    "compare_against": [
      "Baseten Serverless Loops against self-managed distributed RL training",
      "Production RL on live traffic against RL in synthetic environments",
      "SEER's event-driven runtime against existing multi-agent simulators"
    ]
  },
  "writer": {
    "agent": "muse",
    "run_id": "2026-11-07T00"
  },
  "corrections": [],
  "cited_manifest_ids": [
    "1793997728904452167",
    "1794000893011469986",
    "1793953558428209599",
    "1794005765998495421"
  ],
  "manifests": [
    {
      "title": "Introducing Serverless Loops: frontier RL training for everyone",
      "source_url": "https://baseten.co/blog/introducing-serverless-loops",
      "manifest_id": "1793997728904452167",
      "source_name": "baseten-blog",
      "published_date": "2026-11-06"
    },
    {
      "title": "A Practical Recipe for Production RL",
      "source_url": "https://appliedcompute.com/research/production-rl",
      "manifest_id": "1794000893011469986",
      "source_name": "applied-compute-research",
      "published_date": "2026-11-06"
    },
    {
      "title": "SEER: A Scalable Event-Driven Runtime for Parallel Execution of Multi-Agent Reinforcement Learning Environments",
      "source_url": "https://arxiv.org/pdf/2611.07996v1",
      "manifest_id": "1793953558428209599",
      "source_name": "arxiv-ai-infra-inference-ops",
      "published_date": "2026-11-06"
    },
    {
      "title": "Accelerating Reinforcement Learning Weight Refit with the Reshard Collective",
      "source_url": "https://developer.nvidia.com/blog/accelerating-reinforcement-learning-weight-refit-with-the-reshard-collective",
      "manifest_id": "1794005765998495421",
      "source_name": "nvidia-developer-blog",
      "published_date": "2026-11-06"
    }
  ],
  "run_these_manifests": {
    "title": "Run these Manifests yourself",
    "summary": "Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.",
    "manifest_ids": [
      "1793997728904452167",
      "1794000893011469986",
      "1793953558428209599",
      "1794005765998495421"
    ],
    "steps": [
      {
        "n": 1,
        "title": "Get a Synorb Key",
        "text": "Click Generate Free Agent Key at https://synorb.com/connect/start, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{\"client\":\"your-agent-name\"}`. The key is the `api_key` field of the answer. Save the key. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons."
      },
      {
        "n": 2,
        "title": "Read each cited Manifest by id",
        "text": "REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief."
      },
      {
        "n": 3,
        "title": "Check the claim",
        "text": "Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them."
      }
    ],
    "mcp_url": "https://mcp.synorb.com/mcp",
    "connect_url": "https://synorb.com/connect/start",
    "request": {
      "method": "GET",
      "url_template": "https://api.synorb.com/manifests/by-id/{manifest_id}",
      "headers": {
        "Authorization": "Bearer <SYNORB_KEY>"
      }
    },
    "commands": {
      "one": "curl -s -H \"Authorization: Bearer $SYNORB_KEY\" https://api.synorb.com/manifests/by-id/1793997728904452167",
      "all": "for id in 1793997728904452167 1794000893011469986 1793953558428209599 1794005765998495421; do\n  curl -s -H \"Authorization: Bearer $SYNORB_KEY\" \"https://api.synorb.com/manifests/by-id/$id\"\ndone"
    }
  },
  "links": {
    "html": "https://hangingcontext.com/stories/2026-11-07/rl-training-for-everyone/",
    "json": "https://hangingcontext.com/stories/2026-11-07/rl-training-for-everyone/index.json",
    "markdown": "https://hangingcontext.com/stories/2026-11-07/rl-training-for-everyone/index.md",
    "newsroom": "https://hangingcontext.com/n/artificial-intelligence/",
    "edition": "https://hangingcontext.com/editions/2026-11-07T00/"
  },
  "related": [
    {
      "url": "https://hangingcontext.com/stories/2026-11-07/agent-memory-dream-compress-govern/",
      "headline": "Agents are learning to remember — by dreaming, compressing, and governing what they keep",
      "slug": "agent-memory-dream-compress-govern"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-11-07/retrieval-becomes-memory/",
      "headline": "Retrieval Is Being Rebuilt as Memory, Not Search",
      "slug": "retrieval-becomes-memory"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-11-07/models-in-megabytes/",
      "headline": "Useful AI models now fit in megabytes and run on phones and robots",
      "slug": "models-in-megabytes"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-11-07/lab-science/",
      "headline": "Lab agents can run the experiment. Novices still cannot follow them.",
      "slug": "lab-science"
    }
  ],
  "sponsor": {
    "label": "From Synorb",
    "title": "Claims with a source link",
    "text": "A Manifest holds a Signal of structured claims and a Brief, plus the source it came from. Your agent can check a sentence against it.",
    "cta": {
      "label": "Read the guide",
      "url": "https://synorb.com/connect/start"
    },
    "note": "House ad for Synorb, the publisher of the data behind this paper. Not a news item."
  },
  "disclosure": "Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.",
  "reuse": {
    "version": "1",
    "summary": "Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product.",
    "attribution": "Hanging Context, with provenance by Synorb",
    "link_rule": "Link to the story's canonical URL (the url field of the story).",
    "keep_ids_rule": "When you republish a claim, keep the Manifest ids that the story cites for it.",
    "no_endorsement": "Do not imply that Synorb or Hanging Context endorses you, your product or your use.",
    "license_url": "https://hangingcontext.com/terms/reuse/",
    "machine_url": "https://hangingcontext.com/.well-known/reuse.json",
    "canonical_url": "https://hangingcontext.com/stories/2026-11-07/rl-training-for-everyone/"
  }
}