Shared inference is the bottleneck: four preprints attack scheduling, isolation and wasted GPUs

The sharing problem

Serving a model to many tenants at once creates a tension that raw hardware does not resolve. Reusing cached work across requests saves compute, but in a preprint, automatic prefix caching is reported to introduce timing side channels in multi-tenant language-model serving systems. A side channel here means timing differences themselves carry information, so a performance optimisation can become an information leak if tenants share the cache. m:…6050

The usual defence is isolation: keep tenants apart. But isolation throws away reuse, and in a preprint, CacheSolidarity is reported to achieve up to 70% higher cache reuse and 30% lower inference latency than existing user-isolation defences. This suggests the practical question is not whether to isolate, but how selectively to do it, so that tenants keep most of the speed of sharing without keeping the leak. m:…6050

Prefill fights decode

A second conflict sits inside every request. Prefill processes the incoming prompt, while decode generates tokens one at a time, and the two phases want different things from the hardware. In a preprint, decoupling prefill and decode phases in mixture-of-experts inference is reported to allow independent optimisation of scheduling and execution, and expert routing is reported to be predictable in phase-specific ways, which enables effective prefetching and placement strategies. m:…0543

The reported payoff is large. In a preprint, PDScope is reported to achieve 2.5 to 10.3 times higher prefill throughput and 2.1 to 6.1 times higher decode throughput on mainstream mixture-of-experts models. The wide ranges matter: they suggest the gain depends heavily on the model and workload, so a team should expect a direction of improvement rather than a single number it can bank on. m:…0543

Run both phases together, differently

Decoupling is one answer; running both phases together under different rules is another. In a preprint, Conflux Parallelism is reported to allow co-execution of prefill and decode phases with different parallelism strategies within the same batch. That is a subtler idea than simply splitting the work: the batch stays shared, but each phase is parallelised in the way that suits it. m:…6818

In a preprint, ConfluxServe is reported to improve serving goodput by 1.26 to 1.62 times over the strongest baselines. Goodput, the rate of genuinely useful completed work, is the right yardstick here, because a serving system can look busy while completing little. The implication is that scheduling choices, not just model size, decide how much useful work a fleet actually finishes. m:…6818

Multimodal makes it worse

Adding images, audio or video to the same serving system sharpens the scheduling problem, because different modalities stress the hardware differently. In a preprint, disaggregation-based multimodal serving systems are reported to introduce severe GPU underutilisation in moderate-scale deployments. Splitting the system by modality sounds clean, but if one side idles while the other queues, the fleet is paying for capacity it is not using. m:…9791

In a preprint, a modality-aware scheduler called RPS-Serve is reported to increase GPU utilisation by 65% compared with disaggregation-based multimodal systems, and to reduce time to first token by up to 9.3 times on average across multimodal models. Read together, these four preprints point the same way: the next serving gains come from deciding, request by request, what to share, what to separate, and what to schedule first. m:…9791