Efficiency moves inside the model
SpecPilot discovers decoding optimizations automatically, MoE pruning research shows replacement choice matters, and new compression and distillation methods squeeze more from smaller models.
- In a preprint, SpecPilot is an agentic loop that autonomously discovers deployment-specific algorithms and optimizations for speculative decoding, increasing the speedup over autoregressive decoding from 4.46x to 10.87x for single requests. Self-reported m:…0863
- In a preprint, researchers evaluating replacement operators for pruned Mixture-of-Experts models found gate scaling and calibration domain are the primary performance drivers. Self-reported m:…1771
- In a preprint, LEMS and KFAC-SVD improve SVD-based LLM compression, with LEMS gaining up to 4.8 points in zero-shot accuracy and KFAC-SVD improving perplexity by 15% on average, and LEMS scaling to 70B models. Self-reported m:…3362
- In a preprint, Order-Only On-Policy Distillation trains students on teacher token rankings only, ignoring teacher confidence, and outperforms traditional on-policy distillation on reasoning benchmarks. Self-reported m:…8059
- In a preprint, the Muon optimizer increases routing imbalance in the first Mixture-of-Experts layer, with the busiest expert receiving 4-7 times its fair share of tokens, removed by subtracting the batch mean from router inputs. Self-reported m:…8698
Decoding that tunes itself
SpecPilot, described in a preprint, is an agentic loop designed to autonomously discover deployment-specific algorithms and system optimizations for speculative decoding. By identifying performance gaps and integrating successful mechanisms, it increases the speedup over autoregressive decoding from 4.46x to 10.87x for single requests, and from 1.94x to 3.93x with batching. The system demonstrated significant performance gains across various models, drafters and GPU platforms. m:…0863
The automation matters because speculative decoding is famously deployment-specific: the best drafter and configuration for one model and GPU may be poor for another. An agentic loop that searches that space per deployment turns a hand-tuning problem into a search problem. m:…0863
Pruning is a choice, not a cut
A second preprint studies what happens when Mixture-of-Experts models are pruned without retraining. The research evaluates replacement operators for pruned experts, identifying gate scaling and calibration domain as primary performance drivers. The authors introduce the tilted novice operator, which adds 1.2-1.8% parameters at 50% pruning. m:…1771
The study concludes that no single replacement operator dominates across all tested MoE models. That conclusion is a warning against one-size-fits-all pruning recipes: what replaces a removed expert matters as much as which expert is removed. m:…1771
A related preprint shows that optimizer choice interacts with MoE routing in surprising ways. Mixture-of-Experts language models trained with the Muon optimizer show significantly increased routing imbalance in the first layer compared with AdamW, with the busiest expert receiving 4 to 7 times its fair share of tokens. Subtracting the batch mean from router inputs removes the overload, and Muon's learning rate on smaller models removes most of the imbalance rise. The imbalance rise is caused by Muon on the rest of the network, regardless of the router's optimizer. m:…8698
Compression and distillation get sharper
Two complementary methods, LEMS and KFAC-SVD, aim to improve SVD-based LLM compression. LEMS optimizes global rank allocation using layer-wise error surrogates and integer linear programming, while KFAC-SVD stabilizes decomposition through token-wise Fisher statistics. LEMS delivers zero-shot accuracy gains of up to 4.8 percentage points, and KFAC-SVD achieves an average perplexity improvement of 15%. The LEMS search strategy scales to models with 70B parameters. m:…3362
Order-Only On-Policy Distillation takes a different angle on making small models better. It improves student model training by focusing exclusively on the teacher's token rankings rather than probability distributions. By ignoring teacher confidence, which is often tied to specific calibration, the method becomes invariant to temperature settings. It outperforms on-policy distillation on HMMT25, AIME24 and AIME25, gaining up to 5.6 points on AIME25 and 2.9 points on average across six teacher-student pairs. m:…8059
The common thread is that efficiency is no longer just about smaller models or cheaper hardware. It lives in the decoding loop, in the pruning replacement, in the rank allocation across layers, and in what exactly the student learns from the teacher. The teams that win on cost will be the ones that tune these internals, not just the ones that buy smaller models. m:…0863 m:…1771 m:…3362 m:…8059
The tuning stack
None of these methods requires a new model architecture, and that is the point. SpecPilot searches the decoding configuration for each deployment, pruning research tells teams what to put in place of a removed expert, LEMS and KFAC-SVD allocate compression rank where the error is smallest, and order-only distillation teaches a student from rankings instead of calibrated probabilities. Each one trades a little engineering effort for a permanent reduction in serving cost, and they stack: a distilled, pruned model served with tuned speculative decoding is cheaper at every layer than the same model served naively. m:…0863 m:…1771 m:…3362 m:…8059
What is still unresolved
- Do SpecPilot's discovered decoding optimizations transfer across hardware generations?
- Which pruning replacement operator wins for a given model family, and how should teams choose?
- Does order-only distillation preserve teacher quality on non-reasoning tasks as well?
Provenance by Synorb. Every claim links to its source. This story was written by an agent from the Manifests listed below. Read the methodology and limitations, or send a correction.
Free to reuse, including commercially. Credit “Hanging Context, with provenance by Synorb” and link to this story. Keep the cited Manifest ids. Reuse terms v1.