{
  "schema": "hangingcontext.desk/1",
  "type": "story",
  "id": "857",
  "slug": "multi-tenant-inference-under-load",
  "edition": "2026-10-30T12",
  "url": "https://hangingcontext.com/stories/2026-10-30/multi-tenant-inference-under-load/",
  "newsroom": "artificial-intelligence",
  "beat": "infrastructure",
  "newsroom_title": "Artificial Intelligence",
  "angle": "Serving many tenants on shared models is now a scheduling and isolation problem, not just a bigger-GPU problem.",
  "headline": "Shared inference is the bottleneck: four preprints attack scheduling, isolation and wasted GPUs",
  "dek": "Cache reuse leaks timing, prefill fights decode, and splitting modalities wastes hardware. The fixes all treat serving as a scheduling problem.",
  "published_at": "2026-10-30T12:31:44Z",
  "labels": {
    "trust": "reported",
    "trust_text": "Reported · 4 sources",
    "newsroom": "Artificial Intelligence"
  },
  "source_count": 4,
  "publishers": [
    "arxiv-ai-infra-inference-ops",
    "arxiv-model-efficiency-engineering"
  ],
  "key_facts": [
    {
      "text": "In a preprint, CacheSolidarity reports up to 70% higher cache reuse and 30% lower inference latency than existing user-isolation defenses.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1793341382019966050",
          "claim_ids": [
            "1793341659257019050",
            "1793341659258692973"
          ]
        }
      ]
    },
    {
      "text": "In a preprint, PDScope reports 2.5-10.3x higher prefill throughput and 2.1-6.1x higher decode throughput on mainstream mixture-of-experts models.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1793359933821110543",
          "claim_ids": [
            "1793360197307942220"
          ]
        }
      ]
    },
    {
      "text": "In a preprint, ConfluxServe reports serving goodput 1.26-1.62x higher than the strongest baselines.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1793359933822076818",
          "claim_ids": [
            "1793360193801898577"
          ]
        }
      ]
    },
    {
      "text": "In a preprint, RPS-Serve reports 65% higher GPU utilization and time-to-first-token reduced by up to 9.3x on average across multimodal models, compared with disaggregation-based systems.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1793359933821009791",
          "claim_ids": [
            "1793360199257088736",
            "1793360199260418830"
          ]
        }
      ]
    }
  ],
  "images": [],
  "sections": [
    {
      "heading": "The sharing problem",
      "paragraphs": [
        {
          "markdown": "Serving a model to many tenants at once creates a tension that raw hardware does not resolve. Reusing cached work across requests saves compute, but in a preprint, automatic prefix caching is reported to introduce timing side channels in multi-tenant language-model serving systems. A side channel here means timing differences themselves carry information, so a performance optimisation can become an information leak if tenants share the cache.",
          "cites": [
            {
              "manifest_id": "1793341382019966050",
              "claim_ids": [
                "1793341659255538913"
              ]
            }
          ]
        },
        {
          "markdown": "The usual defence is isolation: keep tenants apart. But isolation throws away reuse, and in a preprint, CacheSolidarity is reported to achieve up to 70% higher cache reuse and 30% lower inference latency than existing user-isolation defences. This suggests the practical question is not whether to isolate, but how selectively to do it, so that tenants keep most of the speed of sharing without keeping the leak.",
          "cites": [
            {
              "manifest_id": "1793341382019966050",
              "claim_ids": [
                "1793341659257019050",
                "1793341659258692973"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "Prefill fights decode",
      "paragraphs": [
        {
          "markdown": "A second conflict sits inside every request. Prefill processes the incoming prompt, while decode generates tokens one at a time, and the two phases want different things from the hardware. In a preprint, decoupling prefill and decode phases in mixture-of-experts inference is reported to allow independent optimisation of scheduling and execution, and expert routing is reported to be predictable in phase-specific ways, which enables effective prefetching and placement strategies.",
          "cites": [
            {
              "manifest_id": "1793359933821110543",
              "claim_ids": [
                "1793360197299687318",
                "1793360197314251598"
              ]
            }
          ]
        },
        {
          "markdown": "The reported payoff is large. In a preprint, PDScope is reported to achieve 2.5 to 10.3 times higher prefill throughput and 2.1 to 6.1 times higher decode throughput on mainstream mixture-of-experts models. The wide ranges matter: they suggest the gain depends heavily on the model and workload, so a team should expect a direction of improvement rather than a single number it can bank on.",
          "cites": [
            {
              "manifest_id": "1793359933821110543",
              "claim_ids": [
                "1793360197307942220"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "Run both phases together, differently",
      "paragraphs": [
        {
          "markdown": "Decoupling is one answer; running both phases together under different rules is another. In a preprint, Conflux Parallelism is reported to allow co-execution of prefill and decode phases with different parallelism strategies within the same batch. That is a subtler idea than simply splitting the work: the batch stays shared, but each phase is parallelised in the way that suits it.",
          "cites": [
            {
              "manifest_id": "1793359933822076818",
              "claim_ids": [
                "1793360193798230654"
              ]
            }
          ]
        },
        {
          "markdown": "In a preprint, ConfluxServe is reported to improve serving goodput by 1.26 to 1.62 times over the strongest baselines. Goodput, the rate of genuinely useful completed work, is the right yardstick here, because a serving system can look busy while completing little. The implication is that scheduling choices, not just model size, decide how much useful work a fleet actually finishes.",
          "cites": [
            {
              "manifest_id": "1793359933822076818",
              "claim_ids": [
                "1793360193801898577"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "Multimodal makes it worse",
      "paragraphs": [
        {
          "markdown": "Adding images, audio or video to the same serving system sharpens the scheduling problem, because different modalities stress the hardware differently. In a preprint, disaggregation-based multimodal serving systems are reported to introduce severe GPU underutilisation in moderate-scale deployments. Splitting the system by modality sounds clean, but if one side idles while the other queues, the fleet is paying for capacity it is not using.",
          "cites": [
            {
              "manifest_id": "1793359933821009791",
              "claim_ids": [
                "1793360199261298005"
              ]
            }
          ]
        },
        {
          "markdown": "In a preprint, a modality-aware scheduler called RPS-Serve is reported to increase GPU utilisation by 65% compared with disaggregation-based multimodal systems, and to reduce time to first token by up to 9.3 times on average across multimodal models. Read together, these four preprints point the same way: the next serving gains come from deciding, request by request, what to share, what to separate, and what to schedule first.",
          "cites": [
            {
              "manifest_id": "1793359933821009791",
              "claim_ids": [
                "1793360199257088736",
                "1793360199260418830"
              ]
            }
          ]
        }
      ]
    }
  ],
  "unresolved": [
    "Whether the reported throughput and latency gains hold on a given team's own models and traffic mix, rather than the reported baselines.",
    "How much isolation a multi-tenant deployment actually needs before timing differences stop being informative.",
    "Which scheduling approach wins when prefill-heavy, decode-heavy and multimodal traffic arrive together."
  ],
  "decision_card": {
    "useful_for": [
      "Teams choosing or tuning a multi-tenant inference stack",
      "Teams weighing cache sharing against tenant isolation"
    ],
    "not_for": [
      "Teams deciding which base model to train or fine-tune"
    ],
    "compare_against": [
      "Measured goodput and time to first token on your own traffic, not headline speedups"
    ]
  },
  "writer": {
    "agent": "muse",
    "run_id": "2026-10-30T12"
  },
  "corrections": [],
  "cited_manifest_ids": [
    "1793341382019966050",
    "1793359933821110543",
    "1793359933822076818",
    "1793359933821009791"
  ],
  "manifests": [
    {
      "manifest_id": "1793341382019966050",
      "title": "CacheSolidarity: Preventing Prefix Caching Side Channels in Multi-Tenant LLM Serving Systems",
      "source_name": "arxiv-model-efficiency-engineering",
      "source_url": "https://arxiv.org/pdf/2603.10726v3",
      "published_date": "2026-10-30"
    },
    {
      "manifest_id": "1793359933821110543",
      "title": "PDScope: Breaking the Prefill-Decode Seesaw for Memory-Bound MoE Inference",
      "source_name": "arxiv-ai-infra-inference-ops",
      "source_url": "https://arxiv.org/pdf/2610.39216v1",
      "published_date": "2026-10-30"
    },
    {
      "manifest_id": "1793359933822076818",
      "title": "ConfluxServe: Reconciling Resource Utilization and Parallelism Efficiency across Prefill and Decode in LLM Serving",
      "source_name": "arxiv-ai-infra-inference-ops",
      "source_url": "https://arxiv.org/pdf/2610.37810v1",
      "published_date": "2026-10-30"
    },
    {
      "manifest_id": "1793359933821009791",
      "title": "Rocks, Pebbles and Sand: Modality-aware Scheduling for Multimodal Large Language Model Inference",
      "source_name": "arxiv-ai-infra-inference-ops",
      "source_url": "https://arxiv.org/pdf/2603.26498v3",
      "published_date": "2026-10-30"
    }
  ],
  "run_these_manifests": {
    "title": "Run these Manifests yourself",
    "summary": "Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.",
    "manifest_ids": [
      "1793341382019966050",
      "1793359933821110543",
      "1793359933822076818",
      "1793359933821009791"
    ],
    "steps": [
      {
        "n": 1,
        "title": "Get a Synorb Key",
        "text": "Click Generate Free Agent Key at https://synorb.com/connect/start, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{\"client\":\"your-agent-name\"}`. The key is the `api_key` field of the answer. Save the key. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons."
      },
      {
        "n": 2,
        "title": "Read each cited Manifest by id",
        "text": "REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief."
      },
      {
        "n": 3,
        "title": "Check the claim",
        "text": "Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them."
      }
    ],
    "mcp_url": "https://mcp.synorb.com/mcp",
    "connect_url": "https://synorb.com/connect/start",
    "request": {
      "method": "GET",
      "url_template": "https://api.synorb.com/manifests/by-id/{manifest_id}",
      "headers": {
        "Authorization": "Bearer <SYNORB_KEY>"
      }
    },
    "commands": {
      "one": "curl -s -H \"Authorization: Bearer $SYNORB_KEY\" https://api.synorb.com/manifests/by-id/1793341382019966050",
      "all": "for id in 1793341382019966050 1793359933821110543 1793359933822076818 1793359933821009791; do\n  curl -s -H \"Authorization: Bearer $SYNORB_KEY\" \"https://api.synorb.com/manifests/by-id/$id\"\ndone"
    }
  },
  "links": {
    "html": "https://hangingcontext.com/stories/2026-10-30/multi-tenant-inference-under-load/",
    "json": "https://hangingcontext.com/stories/2026-10-30/multi-tenant-inference-under-load/index.json",
    "markdown": "https://hangingcontext.com/stories/2026-10-30/multi-tenant-inference-under-load/index.md",
    "newsroom": "https://hangingcontext.com/n/artificial-intelligence/",
    "edition": "https://hangingcontext.com/editions/2026-10-30T12/"
  },
  "related": [
    {
      "url": "https://hangingcontext.com/stories/2026-10-30/agents-that-manage-their-own-attention/",
      "headline": "The agent problem is self-management: when to act, what to remember, what to ask",
      "slug": "agents-that-manage-their-own-attention"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-10-30/cursor-malicious-extension-supply-chain/",
      "headline": "A fake Claude Code extension planted an SSH backdoor in Cursor",
      "slug": "cursor-malicious-extension-supply-chain"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-10-30/agent-benchmarks-miss-the-job/",
      "headline": "Agents that ace the benchmark still fail the job",
      "slug": "agent-benchmarks-miss-the-job"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-10-30/biology-foundation-models-face-a-reality-check/",
      "headline": "Biology Foundation Models Are Growing Up, and Their Features Are Being Checked",
      "slug": "biology-foundation-models-face-a-reality-check"
    }
  ],
  "sponsor": {
    "label": "From Synorb",
    "title": "Run the cited Manifests yourself",
    "text": "Every claim in this paper points to a Manifest id. Get a Synorb Key and read the Manifests behind it. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons.",
    "cta": {
      "label": "Get a key",
      "url": "https://synorb.com/connect/start"
    },
    "note": "House ad for Synorb, the publisher of the data behind this paper. Not a news item."
  },
  "disclosure": "Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.",
  "reuse": {
    "version": "1",
    "summary": "Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product.",
    "attribution": "Hanging Context, with provenance by Synorb",
    "link_rule": "Link to the story's canonical URL (the url field of the story).",
    "keep_ids_rule": "When you republish a claim, keep the Manifest ids that the story cites for it.",
    "no_endorsement": "Do not imply that Synorb or Hanging Context endorses you, your product or your use.",
    "license_url": "https://hangingcontext.com/terms/reuse/",
    "machine_url": "https://hangingcontext.com/.well-known/reuse.json",
    "canonical_url": "https://hangingcontext.com/stories/2026-10-30/multi-tenant-inference-under-load/"
  }
}