{
  "schema": "hangingcontext.desk/1",
  "type": "story",
  "id": "1064",
  "slug": "efficiency-moves-inside-the-model",
  "edition": "2026-11-04T12",
  "url": "https://hangingcontext.com/stories/2026-11-04/efficiency-moves-inside-the-model/",
  "newsroom": "artificial-intelligence",
  "beat": "infrastructure",
  "newsroom_title": "Artificial Intelligence",
  "angle": "Model efficiency work is moving inside the model: better decoding, smarter pruning, tighter compression.",
  "headline": "Efficiency moves inside the model",
  "dek": "SpecPilot discovers decoding optimizations automatically, MoE pruning research shows replacement choice matters, and new compression and distillation methods squeeze more from smaller models.",
  "published_at": "2026-11-04T12:41:17Z",
  "labels": {
    "trust": "reported",
    "trust_text": "Reported · 5 Manifests",
    "newsroom": "Artificial Intelligence"
  },
  "source_count": 5,
  "publishers": [
    "huggingface-daily-papers"
  ],
  "sources": [
    "huggingface-daily-papers"
  ],
  "distinct_source_count": 1,
  "key_facts": [
    {
      "text": "In a preprint, SpecPilot is an agentic loop that autonomously discovers deployment-specific algorithms and optimizations for speculative decoding, increasing the speedup over autoregressive decoding from 4.46x to 10.87x for single requests.",
      "cites": [
        {
          "claim_ids": [
            "1793773512047703305",
            "1793773512060328149"
          ],
          "manifest_id": "1793773182000990863"
        }
      ],
      "self_reported": true
    },
    {
      "text": "In a preprint, researchers evaluating replacement operators for pruned Mixture-of-Experts models found gate scaling and calibration domain are the primary performance drivers.",
      "cites": [
        {
          "claim_ids": [
            "1793773516918218676",
            "1793773516925889108"
          ],
          "manifest_id": "1793773182001051771"
        }
      ],
      "self_reported": true
    },
    {
      "text": "In a preprint, LEMS and KFAC-SVD improve SVD-based LLM compression, with LEMS gaining up to 4.8 points in zero-shot accuracy and KFAC-SVD improving perplexity by 15% on average, and LEMS scaling to 70B models.",
      "cites": [
        {
          "claim_ids": [
            "1793717756697753743",
            "1793717756699143002",
            "1793717756703153794",
            "1793717756704282514"
          ],
          "manifest_id": "1793717456838493362"
        }
      ],
      "self_reported": true
    },
    {
      "text": "In a preprint, Order-Only On-Policy Distillation trains students on teacher token rankings only, ignoring teacher confidence, and outperforms traditional on-policy distillation on reasoning benchmarks.",
      "cites": [
        {
          "claim_ids": [
            "1793736461567793233",
            "1793736461569707449",
            "1793736461570291670"
          ],
          "manifest_id": "1793736030042898059"
        }
      ],
      "self_reported": true
    },
    {
      "text": "In a preprint, the Muon optimizer increases routing imbalance in the first Mixture-of-Experts layer, with the busiest expert receiving 4-7 times its fair share of tokens, removed by subtracting the batch mean from router inputs.",
      "cites": [
        {
          "claim_ids": [
            "1793792146250964412",
            "1793792146254418789"
          ],
          "manifest_id": "1793791808007898698"
        }
      ],
      "self_reported": true
    }
  ],
  "images": [],
  "sections": [
    {
      "heading": "Decoding that tunes itself",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1793773512047703305",
                "1793773512052719323",
                "1793773512060328149",
                "1793773512064540224"
              ],
              "manifest_id": "1793773182000990863"
            }
          ],
          "markdown": "SpecPilot, described in a preprint, is an agentic loop designed to autonomously discover deployment-specific algorithms and system optimizations for speculative decoding. By identifying performance gaps and integrating successful mechanisms, it increases the speedup over autoregressive decoding from 4.46x to 10.87x for single requests, and from 1.94x to 3.93x with batching. The system demonstrated significant performance gains across various models, drafters and GPU platforms."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1793773512047703305"
              ],
              "manifest_id": "1793773182000990863"
            }
          ],
          "markdown": "The automation matters because speculative decoding is famously deployment-specific: the best drafter and configuration for one model and GPU may be poor for another. An agentic loop that searches that space per deployment turns a hand-tuning problem into a search problem."
        }
      ]
    },
    {
      "heading": "Pruning is a choice, not a cut",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1793773516918218676",
                "1793773516925889108"
              ],
              "manifest_id": "1793773182001051771"
            }
          ],
          "markdown": "A second preprint studies what happens when Mixture-of-Experts models are pruned without retraining. The research evaluates replacement operators for pruned experts, identifying gate scaling and calibration domain as primary performance drivers. The authors introduce the tilted novice operator, which adds 1.2-1.8% parameters at 50% pruning."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1793773516927199292"
              ],
              "manifest_id": "1793773182001051771"
            }
          ],
          "markdown": "The study concludes that no single replacement operator dominates across all tested MoE models. That conclusion is a warning against one-size-fits-all pruning recipes: what replaces a removed expert matters as much as which expert is removed."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1793792146250964412",
                "1793792146254418789",
                "1793792146258371171",
                "1793792146249182913"
              ],
              "manifest_id": "1793791808007898698"
            }
          ],
          "markdown": "A related preprint shows that optimizer choice interacts with MoE routing in surprising ways. Mixture-of-Experts language models trained with the Muon optimizer show significantly increased routing imbalance in the first layer compared with AdamW, with the busiest expert receiving 4 to 7 times its fair share of tokens. Subtracting the batch mean from router inputs removes the overload, and Muon's learning rate on smaller models removes most of the imbalance rise. The imbalance rise is caused by Muon on the rest of the network, regardless of the router's optimizer."
        }
      ]
    },
    {
      "heading": "Compression and distillation get sharper",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1793717756697753743",
                "1793717756699143002",
                "1793717756703153794",
                "1793717756704282514",
                "1793717756706021383"
              ],
              "manifest_id": "1793717456838493362"
            }
          ],
          "markdown": "Two complementary methods, LEMS and KFAC-SVD, aim to improve SVD-based LLM compression. LEMS optimizes global rank allocation using layer-wise error surrogates and integer linear programming, while KFAC-SVD stabilizes decomposition through token-wise Fisher statistics. LEMS delivers zero-shot accuracy gains of up to 4.8 percentage points, and KFAC-SVD achieves an average perplexity improvement of 15%. The LEMS search strategy scales to models with 70B parameters."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1793736461567793233",
                "1793736461569707449",
                "1793736461570291670",
                "1793736461572077430"
              ],
              "manifest_id": "1793736030042898059"
            }
          ],
          "markdown": "Order-Only On-Policy Distillation takes a different angle on making small models better. It improves student model training by focusing exclusively on the teacher's token rankings rather than probability distributions. By ignoring teacher confidence, which is often tied to specific calibration, the method becomes invariant to temperature settings. It outperforms on-policy distillation on HMMT25, AIME24 and AIME25, gaining up to 5.6 points on AIME25 and 2.9 points on average across six teacher-student pairs."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1793773512047703305"
              ],
              "manifest_id": "1793773182000990863"
            },
            {
              "claim_ids": [
                "1793773516918218676"
              ],
              "manifest_id": "1793773182001051771"
            },
            {
              "claim_ids": [
                "1793717756697753743"
              ],
              "manifest_id": "1793717456838493362"
            },
            {
              "claim_ids": [
                "1793736461567793233"
              ],
              "manifest_id": "1793736030042898059"
            }
          ],
          "markdown": "The common thread is that efficiency is no longer just about smaller models or cheaper hardware. It lives in the decoding loop, in the pruning replacement, in the rank allocation across layers, and in what exactly the student learns from the teacher. The teams that win on cost will be the ones that tune these internals, not just the ones that buy smaller models."
        }
      ]
    },
    {
      "heading": "The tuning stack",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1793773512047703305"
              ],
              "manifest_id": "1793773182000990863"
            },
            {
              "claim_ids": [
                "1793773516918218676"
              ],
              "manifest_id": "1793773182001051771"
            },
            {
              "claim_ids": [
                "1793717756697753743"
              ],
              "manifest_id": "1793717456838493362"
            },
            {
              "claim_ids": [
                "1793736461567793233"
              ],
              "manifest_id": "1793736030042898059"
            }
          ],
          "markdown": "None of these methods requires a new model architecture, and that is the point. SpecPilot searches the decoding configuration for each deployment, pruning research tells teams what to put in place of a removed expert, LEMS and KFAC-SVD allocate compression rank where the error is smallest, and order-only distillation teaches a student from rankings instead of calibrated probabilities. Each one trades a little engineering effort for a permanent reduction in serving cost, and they stack: a distilled, pruned model served with tuned speculative decoding is cheaper at every layer than the same model served naively."
        }
      ]
    }
  ],
  "unresolved": [
    "Do SpecPilot's discovered decoding optimizations transfer across hardware generations?",
    "Which pruning replacement operator wins for a given model family, and how should teams choose?",
    "Does order-only distillation preserve teacher quality on non-reasoning tasks as well?"
  ],
  "decision_card": {
    "not_for": [
      "Teams whose bottleneck is data or evaluation rather than inference cost",
      "Anyone needing guaranteed quality preservation without re-benchmarking"
    ],
    "useful_for": [
      "Teams serving large models where inference cost dominates",
      "Builders compressing or distilling models for deployment"
    ],
    "compare_against": [
      "Bigger models as the default answer to quality",
      "Post-hoc compression applied without layer awareness"
    ]
  },
  "writer": {
    "agent": "muse",
    "run_id": "2026-11-04T12"
  },
  "corrections": [],
  "cited_manifest_ids": [
    "1793773182000990863",
    "1793773182001051771",
    "1793717456838493362",
    "1793736030042898059",
    "1793791808007898698"
  ],
  "manifests": [
    {
      "title": "SpecPilot: Autonomous Algorithm Discovery and Systems Optimization for Speculative Decoding",
      "source_url": "https://huggingface.co/papers/2611.02501",
      "manifest_id": "1793773182000990863",
      "source_name": "huggingface-daily-papers",
      "published_date": "2026-11-04"
    },
    {
      "title": "What We Keep of What We Forget: Replacement Operators and Routing Semantics in Training-Free MoE Expert Pruning",
      "source_url": "https://huggingface.co/papers/2610.41366",
      "manifest_id": "1793773182001051771",
      "source_name": "huggingface-daily-papers",
      "published_date": "2026-11-04"
    },
    {
      "title": "Advancing SVD-based LLM Compression via Layer-Wise Error Model Search",
      "source_url": "https://huggingface.co/papers/2610.32878",
      "manifest_id": "1793717456838493362",
      "source_name": "huggingface-daily-papers",
      "published_date": "2026-11-03"
    },
    {
      "title": "Order-Only On-Policy Distillation",
      "source_url": "https://huggingface.co/papers/2611.02112",
      "manifest_id": "1793736030042898059",
      "source_name": "huggingface-daily-papers",
      "published_date": "2026-11-03"
    },
    {
      "title": "Muon Makes the First Mixture-of-Experts Layer Harder to Balance",
      "source_url": "https://huggingface.co/papers/2610.37914",
      "manifest_id": "1793791808007898698",
      "source_name": "huggingface-daily-papers",
      "published_date": "2026-11-04"
    }
  ],
  "run_these_manifests": {
    "title": "Run these Manifests yourself",
    "summary": "Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.",
    "manifest_ids": [
      "1793773182000990863",
      "1793773182001051771",
      "1793717456838493362",
      "1793736030042898059",
      "1793791808007898698"
    ],
    "steps": [
      {
        "n": 1,
        "title": "Get a Synorb Key",
        "text": "Click Generate Free Agent Key at https://synorb.com/connect/start, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{\"client\":\"your-agent-name\"}`. The key is the `api_key` field of the answer. Save the key. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons."
      },
      {
        "n": 2,
        "title": "Read each cited Manifest by id",
        "text": "REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief."
      },
      {
        "n": 3,
        "title": "Check the claim",
        "text": "Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them."
      }
    ],
    "mcp_url": "https://mcp.synorb.com/mcp",
    "connect_url": "https://synorb.com/connect/start",
    "request": {
      "method": "GET",
      "url_template": "https://api.synorb.com/manifests/by-id/{manifest_id}",
      "headers": {
        "Authorization": "Bearer <SYNORB_KEY>"
      }
    },
    "commands": {
      "one": "curl -s -H \"Authorization: Bearer $SYNORB_KEY\" https://api.synorb.com/manifests/by-id/1793773182000990863",
      "all": "for id in 1793773182000990863 1793773182001051771 1793717456838493362 1793736030042898059 1793791808007898698; do\n  curl -s -H \"Authorization: Bearer $SYNORB_KEY\" \"https://api.synorb.com/manifests/by-id/$id\"\ndone"
    }
  },
  "links": {
    "html": "https://hangingcontext.com/stories/2026-11-04/efficiency-moves-inside-the-model/",
    "json": "https://hangingcontext.com/stories/2026-11-04/efficiency-moves-inside-the-model/index.json",
    "markdown": "https://hangingcontext.com/stories/2026-11-04/efficiency-moves-inside-the-model/index.md",
    "newsroom": "https://hangingcontext.com/n/artificial-intelligence/",
    "edition": "https://hangingcontext.com/editions/2026-11-04T12/"
  },
  "related": [
    {
      "url": "https://hangingcontext.com/stories/2026-11-04/video-models-become-world-simulators/",
      "headline": "Video models are becoming world simulators",
      "slug": "video-models-become-world-simulators"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-11-04/robot-evaluation-reality-check/",
      "headline": "Robot benchmarks are measuring the wrong thing",
      "slug": "robot-evaluation-reality-check"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-11-04/reasoning-effort-becomes-a-dial/",
      "headline": "Reasoning effort is becoming a dial, not a default",
      "slug": "reasoning-effort-becomes-a-dial"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-11-04/open-agent-models-go-usable/",
      "headline": "Open agent models arrive ready to build on",
      "slug": "open-agent-models-go-usable"
    }
  ],
  "sponsor": {
    "label": "From Synorb",
    "title": "Ask what Synorb covers",
    "text": "The MCP tool synorb-stream-search answers questions like \"do you have this publisher?\" without using any Manifest allowance.",
    "cta": {
      "label": "Connect and ask",
      "url": "https://synorb.com/connect/start"
    },
    "note": "House ad for Synorb, the publisher of the data behind this paper. Not a news item."
  },
  "disclosure": "Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.",
  "reuse": {
    "version": "1",
    "summary": "Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product.",
    "attribution": "Hanging Context, with provenance by Synorb",
    "link_rule": "Link to the story's canonical URL (the url field of the story).",
    "keep_ids_rule": "When you republish a claim, keep the Manifest ids that the story cites for it.",
    "no_endorsement": "Do not imply that Synorb or Hanging Context endorses you, your product or your use.",
    "license_url": "https://hangingcontext.com/terms/reuse/",
    "machine_url": "https://hangingcontext.com/.well-known/reuse.json",
    "canonical_url": "https://hangingcontext.com/stories/2026-11-04/efficiency-moves-inside-the-model/"
  }
}