Smaller Footprint, Smarter Scheduling: Inference Gets Cheaper Without Bigger Machines

The laptop result

In a preprint, a system called DeepInfer-Mac executed 750B-class mixture of experts models on 24 gigabytes of unified memory, with a peak process footprint of 16.73 gibibytes. The design separates resident weights from storage-loaded experts and uses next-layer prefetching, which improved throughput by 21.0 percent compared with resident-expert overlap alone. Storage read latency remained the primary bottleneck for token generation, which is the honest limit on the trick. m:…3809

The point is not that a laptop replaces a cluster. The point is that the binding constraint moved. When experts can be paged in with enough foresight, memory capacity stops being the hard wall and storage speed becomes the wall instead. That reframes what edge deployment needs: better prefetching and scheduling, not simply more memory. m:…3809

Adapt per request

A second preprint takes the same adaptive idea to serving. A system called Morph pairs attention and state space layers and uses prefill-oriented gating to select an attention to state space layout for each individual request, with the layout fixed during decoding. Selective batching and hybrid-aware context caching keep batching efficient even as requests diverge. The reported result is serving throughput up to 3.87 times higher than static hybrid baselines. m:…5249

A third preprint moves part of the mixture of experts layer into the network itself. INC-MoE offloads dispatch and combine operations to the network switch and overlaps both collectives with each other and with expert computation. Against the default collective path in vLLM on OLMoE prefill, it reduced time to first token by up to 33 percent and increased throughput by up to 50 percent. Communication, in this design, is work to be overlapped rather than time to be waited out. m:…7721

Prune, schedule, compress

Two more preprints attack the memory wall directly. CRISP co-designs pruning and scheduling for mixture of experts inference on edge servers, reordering neurons by importance so that a single stored expert can support multiple execution widths, and scheduling expert execution by frequency, since expert combinations are highly skewed. It improved throughput by up to 2.1 times on Mixtral-8x7B and DeepSeek-MoE-16B while limiting accuracy loss to 3.6 percent. m:…2679

DCKV takes on the prompt KV cache in hybrid language models with a codec trained end to end, optimizing its transform, quantizer and entropy models against a frozen language model. It achieved more than 55 times compression of KV cache tokens compared with bf16 with under 1.8 points of loss on long-context benchmarks, and more than 100 times compression with under 3.6 points of loss. Long context stops being a memory sentence when the cache itself can be compressed this far. m:…5689

What changes for builders

Across all five preprints, the pattern is consistent: adaptivity beats overprovisioning. Prefetch the next layer, pick the layout per request, overlap communication with computation, schedule experts by frequency, compress the cache against the frozen model. Each technique trades a little complexity for a large reduction in the resource that was actually binding. m:…3809 m:…5249 m:…7721 m:…2679 m:…5689

The limits are stated in the results themselves. Storage latency still bottlenecks laptop-scale mixture of experts inference, and compression still costs accuracy points at the highest ratios. A builder should therefore treat these as a menu of bottlenecks to measure, not as a promise that any model now runs anywhere. Measure which resource binds for a given workload, then pick the adaptation that attacks that resource. m:…3809 m:…5689