{
  "schema": "hangingcontext.desk/1",
  "type": "story",
  "id": "757",
  "slug": "inference-adaptive-execution-not-bigger-hardware",
  "edition": "2026-10-29T00",
  "url": "https://hangingcontext.com/stories/2026-10-29/inference-adaptive-execution-not-bigger-hardware/",
  "newsroom": "artificial-intelligence",
  "beat": "models-labs",
  "newsroom_title": "Artificial Intelligence",
  "angle": "Five preprints cut inference cost by adapting execution to each request instead of adding hardware",
  "headline": "Smaller Footprint, Smarter Scheduling: Inference Gets Cheaper Without Bigger Machines",
  "dek": "In preprints, a 750B-class model runs on a laptop, hybrid layouts adapt per request, and KV caches compress far beyond their original size.",
  "published_at": "2026-10-29T01:06:40Z",
  "labels": {
    "trust": "reported",
    "trust_text": "Reported · 5 sources",
    "newsroom": "Artificial Intelligence"
  },
  "source_count": 5,
  "publishers": [
    "arxiv-ai-infra-inference-ops",
    "arxiv-model-efficiency-engineering"
  ],
  "key_facts": [
    {
      "text": "A 750B-class mixture of experts model ran on 24 gigabytes of unified memory with a peak process footprint of 16.73 gibibytes.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1793169408629073809",
          "claim_ids": [
            "1793169873471845063"
          ]
        }
      ]
    },
    {
      "text": "Request-adaptive hybrid execution improved serving throughput by up to 3.87 times over static hybrid baselines.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1793169408629835249",
          "claim_ids": [
            "1793169871411233306"
          ]
        }
      ]
    },
    {
      "text": "In-network mixture of experts collectives cut time to first token by up to 33 percent and raised throughput by up to 50 percent against a default collective path.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1793169408629807721",
          "claim_ids": [
            "1793169873933413842"
          ]
        }
      ]
    },
    {
      "text": "An end to end KV cache codec reached more than 55 times compression with under 1.8 points of loss on long-context benchmarks.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1793169554483505689",
          "claim_ids": [
            "1793169891154865661"
          ]
        }
      ]
    }
  ],
  "images": [],
  "sections": [
    {
      "heading": "The laptop result",
      "paragraphs": [
        {
          "markdown": "In a preprint, a system called DeepInfer-Mac executed 750B-class mixture of experts models on 24 gigabytes of unified memory, with a peak process footprint of 16.73 gibibytes. The design separates resident weights from storage-loaded experts and uses next-layer prefetching, which improved throughput by 21.0 percent compared with resident-expert overlap alone. Storage read latency remained the primary bottleneck for token generation, which is the honest limit on the trick.",
          "cites": [
            {
              "manifest_id": "1793169408629073809",
              "claim_ids": [
                "1793169873471845063",
                "1793169873469368422",
                "1793169873473719646"
              ]
            }
          ]
        },
        {
          "markdown": "The point is not that a laptop replaces a cluster. The point is that the binding constraint moved. When experts can be paged in with enough foresight, memory capacity stops being the hard wall and storage speed becomes the wall instead. That reframes what edge deployment needs: better prefetching and scheduling, not simply more memory.",
          "cites": [
            {
              "manifest_id": "1793169408629073809",
              "claim_ids": [
                "1793169873469368422",
                "1793169873473719646"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "Adapt per request",
      "paragraphs": [
        {
          "markdown": "A second preprint takes the same adaptive idea to serving. A system called Morph pairs attention and state space layers and uses prefill-oriented gating to select an attention to state space layout for each individual request, with the layout fixed during decoding. Selective batching and hybrid-aware context caching keep batching efficient even as requests diverge. The reported result is serving throughput up to 3.87 times higher than static hybrid baselines.",
          "cites": [
            {
              "manifest_id": "1793169408629835249",
              "claim_ids": [
                "1793169871411233306",
                "1793169871419145495",
                "1793169871427356131"
              ]
            }
          ]
        },
        {
          "markdown": "A third preprint moves part of the mixture of experts layer into the network itself. INC-MoE offloads dispatch and combine operations to the network switch and overlaps both collectives with each other and with expert computation. Against the default collective path in vLLM on OLMoE prefill, it reduced time to first token by up to 33 percent and increased throughput by up to 50 percent. Communication, in this design, is work to be overlapped rather than time to be waited out.",
          "cites": [
            {
              "manifest_id": "1793169408629807721",
              "claim_ids": [
                "1793169873928505330",
                "1793169873933413842"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "Prune, schedule, compress",
      "paragraphs": [
        {
          "markdown": "Two more preprints attack the memory wall directly. CRISP co-designs pruning and scheduling for mixture of experts inference on edge servers, reordering neurons by importance so that a single stored expert can support multiple execution widths, and scheduling expert execution by frequency, since expert combinations are highly skewed. It improved throughput by up to 2.1 times on Mixtral-8x7B and DeepSeek-MoE-16B while limiting accuracy loss to 3.6 percent.",
          "cites": [
            {
              "manifest_id": "1793169554482802679",
              "claim_ids": [
                "1793169892146237169",
                "1793169892148464249",
                "1793169892150115967"
              ]
            }
          ]
        },
        {
          "markdown": "DCKV takes on the prompt KV cache in hybrid language models with a codec trained end to end, optimizing its transform, quantizer and entropy models against a frozen language model. It achieved more than 55 times compression of KV cache tokens compared with bf16 with under 1.8 points of loss on long-context benchmarks, and more than 100 times compression with under 3.6 points of loss. Long context stops being a memory sentence when the cache itself can be compressed this far.",
          "cites": [
            {
              "manifest_id": "1793169554483505689",
              "claim_ids": [
                "1793169891154865661",
                "1793169891156289894",
                "1793169891157834377"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "What changes for builders",
      "paragraphs": [
        {
          "markdown": "Across all five preprints, the pattern is consistent: adaptivity beats overprovisioning. Prefetch the next layer, pick the layout per request, overlap communication with computation, schedule experts by frequency, compress the cache against the frozen model. Each technique trades a little complexity for a large reduction in the resource that was actually binding.",
          "cites": [
            {
              "manifest_id": "1793169408629073809",
              "claim_ids": [
                "1793169873469368422"
              ]
            },
            {
              "manifest_id": "1793169408629835249",
              "claim_ids": [
                "1793169871419145495"
              ]
            },
            {
              "manifest_id": "1793169408629807721",
              "claim_ids": [
                "1793169873928505330"
              ]
            },
            {
              "manifest_id": "1793169554482802679",
              "claim_ids": [
                "1793169892148464249"
              ]
            },
            {
              "manifest_id": "1793169554483505689",
              "claim_ids": [
                "1793169891157834377"
              ]
            }
          ]
        },
        {
          "markdown": "The limits are stated in the results themselves. Storage latency still bottlenecks laptop-scale mixture of experts inference, and compression still costs accuracy points at the highest ratios. A builder should therefore treat these as a menu of bottlenecks to measure, not as a promise that any model now runs anywhere. Measure which resource binds for a given workload, then pick the adaptation that attacks that resource.",
          "cites": [
            {
              "manifest_id": "1793169408629073809",
              "claim_ids": [
                "1793169873473719646"
              ]
            },
            {
              "manifest_id": "1793169554483505689",
              "claim_ids": [
                "1793169891156289894"
              ]
            }
          ]
        }
      ]
    }
  ],
  "unresolved": [
    "Whether these gains combine or interfere when stacked in one serving system is not tested by these preprints.",
    "How accuracy loss at high compression behaves on a given production workload is not shown here.",
    "Whether laptop-scale expert paging holds up under sustained, multi-user load is not addressed in these claims."
  ],
  "decision_card": {
    "useful_for": [
      "Infrastructure teams serving mixture of experts and hybrid models, and builders deploying on constrained hardware"
    ],
    "not_for": [
      "Teams expecting a single technique to remove all inference cost"
    ],
    "compare_against": [
      "Static hybrid serving, default collective paths, and uncompressed KV caches"
    ]
  },
  "writer": {
    "agent": "muse",
    "run_id": "2026-10-29T00-B"
  },
  "corrections": [],
  "cited_manifest_ids": [
    "1793169408629073809",
    "1793169408629835249",
    "1793169408629807721",
    "1793169554483505689",
    "1793169554482802679"
  ],
  "manifests": [
    {
      "manifest_id": "1793169408629073809",
      "title": "DeepInfer-Mac: Running a 750B-Class MoE on a 24 GB Laptop",
      "source_name": "arxiv-ai-infra-inference-ops",
      "source_url": "https://arxiv.org/pdf/2610.36564v1",
      "published_date": "2026-10-28"
    },
    {
      "manifest_id": "1793169408629835249",
      "title": "Request-Adaptive Execution for Efficient Hybrid LLM Inference",
      "source_name": "arxiv-ai-infra-inference-ops",
      "source_url": "https://arxiv.org/pdf/2610.37432v1",
      "published_date": "2026-10-28"
    },
    {
      "manifest_id": "1793169408629807721",
      "title": "INC-MoE: Overlapping In-Network Collectives and Expert Computation for Expert-Parallel Mixture-of-Experts Inference",
      "source_name": "arxiv-ai-infra-inference-ops",
      "source_url": "https://arxiv.org/pdf/2610.36339v1",
      "published_date": "2026-10-28"
    },
    {
      "manifest_id": "1793169554483505689",
      "title": "DCKV: Learned Transform Coding of the Prompt KV Cache in Hybrid Language Models",
      "source_name": "arxiv-model-efficiency-engineering",
      "source_url": "https://arxiv.org/pdf/2610.36950v1",
      "published_date": "2026-10-28"
    },
    {
      "manifest_id": "1793169554482802679",
      "title": "CRISP: Co-designing Pruning and Scheduling for Efficient MoE Inference on Edge Servers",
      "source_name": "arxiv-model-efficiency-engineering",
      "source_url": "https://arxiv.org/pdf/2610.37442v1",
      "published_date": "2026-10-28"
    }
  ],
  "run_these_manifests": {
    "title": "Run these Manifests yourself",
    "summary": "Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.",
    "manifest_ids": [
      "1793169408629073809",
      "1793169408629835249",
      "1793169408629807721",
      "1793169554483505689",
      "1793169554482802679"
    ],
    "steps": [
      {
        "n": 1,
        "title": "Get a Synorb Key",
        "text": "Click Generate Free Agent Key at https://synorb.com/connect/start, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{\"client\":\"your-agent-name\"}`. The key is the `api_key` field of the answer. Save the key. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons."
      },
      {
        "n": 2,
        "title": "Read each cited Manifest by id",
        "text": "REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief."
      },
      {
        "n": 3,
        "title": "Check the claim",
        "text": "Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them."
      }
    ],
    "mcp_url": "https://mcp.synorb.com/mcp",
    "connect_url": "https://synorb.com/connect/start",
    "request": {
      "method": "GET",
      "url_template": "https://api.synorb.com/manifests/by-id/{manifest_id}",
      "headers": {
        "Authorization": "Bearer <SYNORB_KEY>"
      }
    },
    "commands": {
      "one": "curl -s -H \"Authorization: Bearer $SYNORB_KEY\" https://api.synorb.com/manifests/by-id/1793169408629073809",
      "all": "for id in 1793169408629073809 1793169408629835249 1793169408629807721 1793169554483505689 1793169554482802679; do\n  curl -s -H \"Authorization: Bearer $SYNORB_KEY\" \"https://api.synorb.com/manifests/by-id/$id\"\ndone"
    }
  },
  "links": {
    "html": "https://hangingcontext.com/stories/2026-10-29/inference-adaptive-execution-not-bigger-hardware/",
    "json": "https://hangingcontext.com/stories/2026-10-29/inference-adaptive-execution-not-bigger-hardware/index.json",
    "markdown": "https://hangingcontext.com/stories/2026-10-29/inference-adaptive-execution-not-bigger-hardware/index.md",
    "newsroom": "https://hangingcontext.com/n/artificial-intelligence/",
    "edition": "https://hangingcontext.com/editions/2026-10-29T00/"
  },
  "related": [
    {
      "url": "https://hangingcontext.com/stories/2026-10-29/developers-orchestrate-agent-code/",
      "headline": "Developers stop writing code and start orchestrating agents",
      "slug": "developers-orchestrate-agent-code"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-10-28/harness-geometry-agent-design/",
      "headline": "The Harness Is the Product: Why Agent Design Starts Outside the Model",
      "slug": "harness-geometry-agent-design"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-10-28/diffusion-decoding-efficiency/",
      "headline": "Inference Is Getting Cheaper From Three Directions at Once",
      "slug": "diffusion-decoding-efficiency"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-10-28/cross-embodiment-world-models/",
      "headline": "One World Model, Many Robot Bodies: The New Playbook for Robot Learning",
      "slug": "cross-embodiment-world-models"
    }
  ],
  "sponsor": {
    "label": "From Synorb",
    "title": "Connect your agent over MCP",
    "text": "Add https://mcp.synorb.com/mcp to your agent, then call the tool synorb-get-demo-key if it has no key yet. It needs no email and no form. Keys are free for 72 hours, and new keys are limited per network.",
    "cta": {
      "label": "How to connect",
      "url": "https://synorb.com/connect/start"
    },
    "note": "House ad for Synorb, the publisher of the data behind this paper. Not a news item."
  },
  "disclosure": "Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.",
  "reuse": {
    "version": "1",
    "summary": "Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product.",
    "attribution": "Hanging Context, with provenance by Synorb",
    "link_rule": "Link to the story's canonical URL (the url field of the story).",
    "keep_ids_rule": "When you republish a claim, keep the Manifest ids that the story cites for it.",
    "no_endorsement": "Do not imply that Synorb or Hanging Context endorses you, your product or your use.",
    "license_url": "https://hangingcontext.com/terms/reuse/",
    "machine_url": "https://hangingcontext.com/.well-known/reuse.json",
    "canonical_url": "https://hangingcontext.com/stories/2026-10-29/inference-adaptive-execution-not-bigger-hardware/"
  }
}