The MoE tax is shrinking: four preprints attack sparse-model inference costs

Do you need all those experts?

Mixture-of-experts won on the promise that sparse models give dense-model quality at a fraction of the compute per token. The bill arrived anyway: expert redundancy across layers, KV caches that balloon with sparse attention, and offloading bottlenecks. In a preprint posted this week, the authors behind Meta Expert ask the bluntest question — do you need all those experts? Their answer is no: sharing meta-experts across layers reduces redundancy and improves efficiency versus layer-specific local experts, matching vanilla MoE performance with only 70-80% of the experts and comparable loss with 75-90% as many parameters. The savings grow as models get deeper and larger, which is exactly where the economics hurt most. m:…0425

The redundancy the paper removes is real, not cosmetic: models incorporating Meta Expert show lower parameter redundancy, lower cross-entropy loss, and higher downstream task accuracy than vanilla MoE, and the shared experts provide hidden states with more flexible residual pathways, improving information retrieval efficiency inside the model. Sharing experts is not just a compression trick — it changes how information routes through the network. m:…0425

That is a training-time fix. For the serving stack, a second preprint this week — Specter — attacks the other end: the PCIe I/O bottleneck that appears when experts are offloaded to host memory, because routing is data-dependent and you never know which experts will fire. Specter's trick is lightweight quantized draft models that preserve high-fidelity expert activation patterns, enabling accurate multi-step lookahead so the system can move experts before they are needed. The result: an average 1.50x and up to 1.88x speedup in time per output token over state-of-the-art systems. m:…4866

Compress the cache, keep the routing

The KV cache is the third front. In a third preprint this week, MoEZip proposes routing-aware KV cache compression: instead of compressing blindly, it uses the Fisher information matrix of the routing distributions, which preserves expert selection quality — the thing that actually determines output quality in sparse models. At a 10% KV retention ratio it scores 2.3x and 4.9x higher on LongBench and RULER respectively than state-of-the-art baselines. For long-context MoE serving, where the cache dominates memory cost, that is a meaningful margin. m:…8662

The fourth preprint, DS-MoE, goes after the selection rule itself. Existing expert selection heuristics mostly rely on top-k ranking, which evaluates each expert in isolation and ignores inter-expert dependencies. The authors show expert combinations exhibit functional duality — redundancy from overlapping representations, synergy from complementary error cancellation — and decouple the two with a second-order Taylor expansion of loss degradation. Their majorization-minimization algorithm for choosing expert subsets comes with provable guarantees. This is the kind of result that migrates into serving frameworks within a year. m:…6257

What this means for margins

Taken together, the four papers describe a MoE stack where the expert count, the cache, the offload path, and the selection rule are all being re-engineered at once. For anyone serving open-weights MoE models — or pricing API margins against them — the implication is that sparse-model inference cost has real headroom left. The labs training 70%-of-the-experts models and the teams serving them at 10% KV retention will be quoting different unit economics within a year. The moat for MoE serving is not the model weights; it is the stack around them. m:…0425 m:…4866 m:…8662 m:…6257