---
title: Efficiency moves inside the model
description: SpecPilot discovers decoding optimizations automatically, MoE pruning research shows replacement choice matters, and new compression and distillation methods squeeze more from smaller models.
canonical: https://hangingcontext.com/stories/2026-11-04/efficiency-moves-inside-the-model/
last-updated: 2026-11-04
url: https://hangingcontext.com/stories/2026-11-04/efficiency-moves-inside-the-model/
published: 2026-11-04T12:41:17Z
edition: 2026-11-04T12
newsroom: artificial-intelligence
beat: infrastructure
trust: reported
source_count: 5
cited_manifest_ids: ['1793773182000990863', '1793773182001051771', '1793717456838493362', '1793736030042898059', '1793791808007898698']
---

# Efficiency moves inside the model

*SpecPilot discovers decoding optimizations automatically, MoE pruning research shows replacement choice matters, and new compression and distillation methods squeeze more from smaller models.*

Reported · 5 Manifests · Artificial Intelligence · Published 2026-11-04 12:41 UTC

## Key facts

- In a preprint, SpecPilot is an agentic loop that autonomously discovers deployment-specific algorithms and optimizations for speculative decoding, increasing the speedup over autoregressive decoding from 4.46x to 10.87x for single requests. (self-reported) [cites: 1793773182000990863]
- In a preprint, researchers evaluating replacement operators for pruned Mixture-of-Experts models found gate scaling and calibration domain are the primary performance drivers. (self-reported) [cites: 1793773182001051771]
- In a preprint, LEMS and KFAC-SVD improve SVD-based LLM compression, with LEMS gaining up to 4.8 points in zero-shot accuracy and KFAC-SVD improving perplexity by 15% on average, and LEMS scaling to 70B models. (self-reported) [cites: 1793717456838493362]
- In a preprint, Order-Only On-Policy Distillation trains students on teacher token rankings only, ignoring teacher confidence, and outperforms traditional on-policy distillation on reasoning benchmarks. (self-reported) [cites: 1793736030042898059]
- In a preprint, the Muon optimizer increases routing imbalance in the first Mixture-of-Experts layer, with the busiest expert receiving 4-7 times its fair share of tokens, removed by subtracting the batch mean from router inputs. (self-reported) [cites: 1793791808007898698]

## Decoding that tunes itself

SpecPilot, described in a preprint, is an agentic loop designed to autonomously discover deployment-specific algorithms and system optimizations for speculative decoding. By identifying performance gaps and integrating successful mechanisms, it increases the speedup over autoregressive decoding from 4.46x to 10.87x for single requests, and from 1.94x to 3.93x with batching. The system demonstrated significant performance gains across various models, drafters and GPU platforms. [cites: 1793773182000990863]

The automation matters because speculative decoding is famously deployment-specific: the best drafter and configuration for one model and GPU may be poor for another. An agentic loop that searches that space per deployment turns a hand-tuning problem into a search problem. [cites: 1793773182000990863]

## Pruning is a choice, not a cut

A second preprint studies what happens when Mixture-of-Experts models are pruned without retraining. The research evaluates replacement operators for pruned experts, identifying gate scaling and calibration domain as primary performance drivers. The authors introduce the tilted novice operator, which adds 1.2-1.8% parameters at 50% pruning. [cites: 1793773182001051771]

> **From Synorb** — Ask what Synorb covers. The MCP tool synorb-stream-search answers questions like "do you have this publisher?" without using any Manifest allowance. [Connect and ask](https://synorb.com/connect/start)


The study concludes that no single replacement operator dominates across all tested MoE models. That conclusion is a warning against one-size-fits-all pruning recipes: what replaces a removed expert matters as much as which expert is removed. [cites: 1793773182001051771]

A related preprint shows that optimizer choice interacts with MoE routing in surprising ways. Mixture-of-Experts language models trained with the Muon optimizer show significantly increased routing imbalance in the first layer compared with AdamW, with the busiest expert receiving 4 to 7 times its fair share of tokens. Subtracting the batch mean from router inputs removes the overload, and Muon's learning rate on smaller models removes most of the imbalance rise. The imbalance rise is caused by Muon on the rest of the network, regardless of the router's optimizer. [cites: 1793791808007898698]

## Compression and distillation get sharper

Two complementary methods, LEMS and KFAC-SVD, aim to improve SVD-based LLM compression. LEMS optimizes global rank allocation using layer-wise error surrogates and integer linear programming, while KFAC-SVD stabilizes decomposition through token-wise Fisher statistics. LEMS delivers zero-shot accuracy gains of up to 4.8 percentage points, and KFAC-SVD achieves an average perplexity improvement of 15%. The LEMS search strategy scales to models with 70B parameters. [cites: 1793717456838493362]

Order-Only On-Policy Distillation takes a different angle on making small models better. It improves student model training by focusing exclusively on the teacher's token rankings rather than probability distributions. By ignoring teacher confidence, which is often tied to specific calibration, the method becomes invariant to temperature settings. It outperforms on-policy distillation on HMMT25, AIME24 and AIME25, gaining up to 5.6 points on AIME25 and 2.9 points on average across six teacher-student pairs. [cites: 1793736030042898059]

The common thread is that efficiency is no longer just about smaller models or cheaper hardware. It lives in the decoding loop, in the pruning replacement, in the rank allocation across layers, and in what exactly the student learns from the teacher. The teams that win on cost will be the ones that tune these internals, not just the ones that buy smaller models. [cites: 1793773182000990863, 1793773182001051771, 1793717456838493362, 1793736030042898059]

## The tuning stack

None of these methods requires a new model architecture, and that is the point. SpecPilot searches the decoding configuration for each deployment, pruning research tells teams what to put in place of a removed expert, LEMS and KFAC-SVD allocate compression rank where the error is smallest, and order-only distillation teaches a student from rankings instead of calibrated probabilities. Each one trades a little engineering effort for a permanent reduction in serving cost, and they stack: a distilled, pruned model served with tuned speculative decoding is cheaper at every layer than the same model served naively. [cites: 1793773182000990863, 1793773182001051771, 1793717456838493362, 1793736030042898059]

## What is still unresolved

- Do SpecPilot's discovered decoding optimizations transfer across hardware generations?
- Which pruning replacement operator wins for a given model family, and how should teams choose?
- Does order-only distillation preserve teacher quality on non-reasoning tasks as well?

## Decision card

- **Useful for:** Teams serving large models where inference cost dominates; Builders compressing or distilling models for deployment
- **Not for:** Teams whose bottleneck is data or evaluation rather than inference cost; Anyone needing guaranteed quality preservation without re-benchmarking
- **Compare against:** Bigger models as the default answer to quality; Post-hoc compression applied without layer awareness

## Provenance ledger

| Manifest id | Title | Source | Claims used | Published |
|---|---|---|---|---|
| `1793773182000990863` | SpecPilot: Autonomous Algorithm Discovery and Systems Optimization for Speculative Decoding | [huggingface-daily-papers](https://huggingface.co/papers/2611.02501) | 1793773512047703305, 1793773512060328149, 1793773512052719323, 1793773512064540224 | 11-04 00:00 |
| `1793773182001051771` | What We Keep of What We Forget: Replacement Operators and Routing Semantics in Training-Free MoE Expert Pruning | [huggingface-daily-papers](https://huggingface.co/papers/2610.41366) | 1793773516918218676, 1793773516925889108, 1793773516927199292 | 11-04 00:00 |
| `1793717456838493362` | Advancing SVD-based LLM Compression via Layer-Wise Error Model Search | [huggingface-daily-papers](https://huggingface.co/papers/2610.32878) | 1793717756697753743, 1793717756699143002, 1793717756703153794, 1793717756704282514, 1793717756706021383 | 11-03 00:00 |
| `1793736030042898059` | Order-Only On-Policy Distillation | [huggingface-daily-papers](https://huggingface.co/papers/2611.02112) | 1793736461567793233, 1793736461569707449, 1793736461570291670, 1793736461572077430 | 11-03 00:00 |
| `1793791808007898698` | Muon Makes the First Mixture-of-Experts Layer Harder to Balance | [huggingface-daily-papers](https://huggingface.co/papers/2610.37914) | 1793792146250964412, 1793792146254418789, 1793792146258371171, 1793792146249182913 | 11-04 00:00 |

## Run these Manifests yourself

Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.

1. **Get a Synorb Key.** Click Generate Free Agent Key at https://synorb.com/connect/start, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{"client":"your-agent-name"}`. The key is the `api_key` field of the answer. Save the key. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons.
2. **Read each cited Manifest by id.** REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief.
3. **Check the claim.** Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them.

Cited Manifest ids:

- `1793773182000990863`
- `1793773182001051771`
- `1793717456838493362`
- `1793736030042898059`
- `1793791808007898698`

```sh
for id in 1793773182000990863 1793773182001051771 1793717456838493362 1793736030042898059 1793791808007898698; do
  curl -s -H "Authorization: Bearer $SYNORB_KEY" "https://api.synorb.com/manifests/by-id/$id"
done
```

> **From Synorb** — Run the cited Manifests yourself. Every claim in this paper points to a Manifest id. Get a Synorb Key and read the Manifests behind it. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons. [Get a key](https://synorb.com/connect/start)

## Related stories

- [Video models are becoming world simulators](https://hangingcontext.com/stories/2026-11-04/video-models-become-world-simulators/)
- [Robot benchmarks are measuring the wrong thing](https://hangingcontext.com/stories/2026-11-04/robot-evaluation-reality-check/)
- [Reasoning effort is becoming a dial, not a default](https://hangingcontext.com/stories/2026-11-04/reasoning-effort-becomes-a-dial/)
- [Open agent models arrive ready to build on](https://hangingcontext.com/stories/2026-11-04/open-agent-models-go-usable/)

## Links

- HTML: https://hangingcontext.com/stories/2026-11-04/efficiency-moves-inside-the-model/
- JSON: https://hangingcontext.com/stories/2026-11-04/efficiency-moves-inside-the-model/index.json
- Markdown: https://hangingcontext.com/stories/2026-11-04/efficiency-moves-inside-the-model/index.md

Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.

---
Reuse (terms v1): Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product. Credit: "Hanging Context, with provenance by Synorb", with a link to https://hangingcontext.com/stories/2026-11-04/efficiency-moves-inside-the-model/. Terms: https://hangingcontext.com/terms/reuse/