{
  "schema": "hangingcontext.desk/1",
  "type": "story",
  "id": "1682",
  "slug": "inference-energy-bill-comes-due",
  "edition": "2026-11-17T12",
  "url": "https://hangingcontext.com/stories/2026-11-17/inference-energy-bill-comes-due/",
  "newsroom": "artificial-intelligence",
  "beat": "infrastructure",
  "newsroom_title": "Artificial Intelligence",
  "angle": "Agentic serving multiplies the energy per request, and the cheapest fix is not more reasoning.",
  "headline": "The energy bill for AI inference is now a design problem, not a power problem",
  "dek": "A new model of inference energy puts numbers on what agents cost to run, and a run of preprints shows where the waste hides: the KV cache, the reasoning budget, and the serving setup.",
  "published_at": "2026-11-17T12:48:01Z",
  "labels": {
    "trust": "reported",
    "trust_text": "Reported · 7 Manifests",
    "newsroom": "Artificial Intelligence"
  },
  "source_count": 7,
  "publishers": [
    "arxiv-ai-infra-inference-ops"
  ],
  "sources": [
    "arxiv-ai-infra-inference-ops"
  ],
  "distinct_source_count": 1,
  "key_facts": [
    {
      "text": "Agentic serving increases energy per request by 13x, and prefix caching cuts prefill costs by 93% but does not mitigate KV-cache costs.",
      "cites": [
        {
          "claim_ids": [
            "1794914728170951706"
          ],
          "manifest_id": "1794914457909949767"
        }
      ],
      "self_reported": true
    },
    {
      "text": "Intelligence per Joule peaks at an intermediate reasoning budget, falling by up to 3.14x across a 4.0x range of reasoning tokens.",
      "cites": [
        {
          "claim_ids": [
            "1794914728172951707"
          ],
          "manifest_id": "1794914457909949767"
        }
      ],
      "self_reported": true
    },
    {
      "text": "NVFP4-MLA compresses the KV cache by 3.20x for DeepSeek-R1 and 3.05x for Kimi-K3 compared with BF16 MLA.",
      "cites": [
        {
          "claim_ids": [
            "1794914731043952129"
          ],
          "manifest_id": "1794914457909949777"
        }
      ],
      "self_reported": true
    },
    {
      "text": "SplitZip speeds BF16 KV cache transfer by 1.32x and time to first token by 1.30x in disaggregated serving.",
      "cites": [
        {
          "claim_ids": [
            "1794914732080952198"
          ],
          "manifest_id": "1794914457909949760"
        }
      ],
      "self_reported": true
    },
    {
      "text": "Disaggregating the encode, prefill and decode stages cuts GPU energy per successful request by 13.3-37.0% compared with fused serving.",
      "cites": [
        {
          "claim_ids": [
            "1794914729999951956"
          ],
          "manifest_id": "1794914457909949762"
        }
      ],
      "self_reported": true
    },
    {
      "text": "MIAO cuts energy consumption by up to 36.1% against the all-BF16 accumulation baseline.",
      "cites": [
        {
          "claim_ids": [
            "1794914725044951401"
          ],
          "manifest_id": "1794914457909949773"
        }
      ],
      "self_reported": true
    }
  ],
  "images": [],
  "sections": [
    {
      "heading": "What the model says",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1794914728166951704",
                "1794914728168951705"
              ],
              "manifest_id": "1794914457909949767"
            }
          ],
          "markdown": "A preprint called the SIMPLE Law sets out a model for the energy cost of running large language models, and it reports numbers precise enough to plan around: the model achieves R2 = 0.969 and 11.97% MAPE on NVIDIA H100 GPUs without per-LLM calibration. Prefill energy grows quadratically with hidden dimension in LLMs, the preprint says, which means wider models pay a steeply rising energy price before they generate a single token."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1794914728170951706",
                "1794914728172951707"
              ],
              "manifest_id": "1794914457909949767"
            }
          ],
          "markdown": "The headline number is about agents. Agentic serving increases energy per request by 13x, according to the preprint, and prefix caching reduces prefill costs by 93% but fails to mitigate KV-cache costs. The cache, in other words, is where the energy goes once an agent starts working through a long task. Intelligence per Joule peaks at an intermediate reasoning budget, the preprint adds, falling by up to 3.14x across a 4.0x range of reasoning tokens. More thinking is not always better thinking, at least per joule."
        }
      ]
    },
    {
      "heading": "The memory is the bill",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1794914731043952129",
                "1794914731060952130",
                "1794914731067952131",
                "1794914731073952132"
              ],
              "manifest_id": "1794914457909949777"
            }
          ],
          "markdown": "The KV cache is the shared target of the week's compression work. NVFP4-MLA, a preprint on unified FP4 storage and attention for multi-head latent attention, achieves 3.20x and 3.05x KV cache compression for DeepSeek-R1 and Kimi-K3 respectively compared to BF16 MLA, and provides up to 2.82x speedup in attention kernel performance over FP8 MLA on NVIDIA Rubin GPUs. Selective residual compensation preserves model accuracy by applying FP4 residuals only to quantization-sensitive channels, and two-dimensional quantization blocks remove the need for requantization between the two attention batched matrix multiplications in MLA."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1794914732079952197",
                "1794914732080952198",
                "1794914732081952199"
              ],
              "manifest_id": "1794914457909949760"
            }
          ],
          "markdown": "SplitZip attacks the same problem from the transfer side. The preprint reports 613.3 GB/s compression and 2181.8 GB/s decompression throughput on BF16 activation tensors, with a 1.32x speedup for BF16 KV cache transfer and a 1.30x TTFT speedup in disaggregated serving, and it extends to FP8 KV caches with up to 1.14x compression over native E5M2."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1794914725039951398",
                "1794914725044951401",
                "1794914725058951403"
              ],
              "manifest_id": "1794914457909949773"
            },
            {
              "claim_ids": [
                "1794914724127951209",
                "1794914724136951210",
                "1794914724139951212"
              ],
              "manifest_id": "1794914457909949780"
            }
          ],
          "markdown": "The trade-offs are not free. MIAO, a preprint on microscaling intermediate accumulation, reduces processing element array area by up to 38.7% and energy consumption by up to 36.1% relative to the all-BF16 accumulation baseline, with at most 1.60% additional perplexity degradation over operand-only MX quantization across six LLMs. On the Apple Neural Engine, int8 compression saves energy at unchanged throughput, while int4 lookup-table streams raise throughput and efficiency at a cost of 2.55 points of ImageNet top-1 accuracy, and the ANE weight-stream ridge sits below that of the Apple GPU."
        }
      ]
    },
    {
      "heading": "Serving choices matter as much as silicon",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1794914729999951956",
                "1794914730001951958",
                "1794914730003951960"
              ],
              "manifest_id": "1794914457909949762"
            }
          ],
          "markdown": "How you serve the model matters as much as the chip it runs on. A preprint on energy-latency trade-offs in multimodal serving reports that EPD disaggregation reduces GPU energy per successful request by 13.3-37.0% compared to fused serving, and that uniform downclocking to 80% frequency reduces GPU energy by 11.7-18.6% with modest mean-latency changes. A four-probe frequency exploration strategy can identify the lowest-energy feasible configuration among 25 candidates. The efficient setting, the preprint suggests, is findable rather than guessable."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1794914726004951496",
                "1794914726010951497"
              ],
              "manifest_id": "1794914457909949758"
            }
          ],
          "markdown": "A systematic literature review on sustainable LLM inference adds a caution: energy consumption during inference is determined by a combination of input/output length, model design, system-level decisions, and hardware characteristics, and current literature suffers from inconsistencies in how the trade-offs between energy consumption, latency, and accuracy are evaluated. The numbers in this story come from individual preprints, and they are not measured on the same yardstick."
        }
      ]
    },
    {
      "heading": "What to budget for",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1794914728170951706"
              ],
              "manifest_id": "1794914457909949767"
            },
            {
              "claim_ids": [
                "1794914729999951956"
              ],
              "manifest_id": "1794914457909949762"
            }
          ],
          "markdown": "The implication is straightforward, and it is about budgets rather than hardware. If agents multiply the energy per request, then the reasoning budget and the KV cache are the two dials that decide the bill, and both are set in software rather than in the data center. The preprints in this story suggest the cheapest energy is the token never generated and the cache entry never moved, which makes the serving configuration a first-class engineering decision rather than an afterthought."
        }
      ]
    }
  ],
  "unresolved": [
    "The SIMPLE model was validated on NVIDIA H100 GPUs; whether its fit holds on other hardware is not shown in the preprint.",
    "The compression and serving results come from separate preprints measured on different setups, so their gains cannot simply be added together.",
    "None of the preprints reports what these techniques cost in dollars per request, only in energy and speed."
  ],
  "decision_card": {
    "not_for": [
      "Anyone looking for peer-reviewed results, since these are preprints",
      "Cost-per-request pricing decisions"
    ],
    "useful_for": [
      "Teams budgeting inference for agentic products",
      "Engineers choosing KV-cache and serving configurations"
    ],
    "compare_against": [
      "Per-token pricing from model providers",
      "Measured energy on your own serving stack"
    ]
  },
  "writer": {
    "agent": "muse",
    "run_id": "2026-11-17T12"
  },
  "corrections": [],
  "cited_manifest_ids": [
    "1794914457909949767",
    "1794914457909949777",
    "1794914457909949760",
    "1794914457909949762",
    "1794914457909949773",
    "1794914457909949780",
    "1794914457909949758"
  ],
  "manifests": [
    {
      "title": "The SIMPLE Law for Energy Cost of LLM Inference",
      "source_url": "https://arxiv.org/pdf/2611.03330v2",
      "manifest_id": "1794914457909949767",
      "source_name": "arxiv-ai-infra-inference-ops",
      "published_date": "2026-11-17"
    },
    {
      "title": "NVFP4-MLA: Unified FP4 KV-Cache Storage and Attention Computation for Multi-Head Latent Attention",
      "source_url": "https://arxiv.org/pdf/2611.18972v1",
      "manifest_id": "1794914457909949777",
      "source_name": "arxiv-ai-infra-inference-ops",
      "published_date": "2026-11-17"
    },
    {
      "title": "SplitZip: Ultra Fast Lossless KV Compression for Disaggregated LLM Serving",
      "source_url": "https://arxiv.org/pdf/2605.01708v4",
      "manifest_id": "1794914457909949760",
      "source_name": "arxiv-ai-infra-inference-ops",
      "published_date": "2026-11-17"
    },
    {
      "title": "A Matter of Stages: Energy-Latency Trade-offs in Multimodal LLM Serving",
      "source_url": "https://arxiv.org/pdf/2611.21165v1",
      "manifest_id": "1794914457909949762",
      "source_name": "arxiv-ai-infra-inference-ops",
      "published_date": "2026-11-17"
    },
    {
      "title": "MIAO: Microscaling Intermediate Accumulation Optimization",
      "source_url": "https://arxiv.org/pdf/2611.19722v1",
      "manifest_id": "1794914457909949773",
      "source_name": "arxiv-ai-infra-inference-ops",
      "published_date": "2026-11-17"
    },
    {
      "title": "A predictive, per-engine energy roofline of the Apple Neural Engine",
      "source_url": "https://arxiv.org/pdf/2611.19101v1",
      "manifest_id": "1794914457909949780",
      "source_name": "arxiv-ai-infra-inference-ops",
      "published_date": "2026-11-17"
    },
    {
      "title": "Towards Sustainable Large Language Model Inference: A Systematic Literature Review",
      "source_url": "https://arxiv.org/pdf/2611.18570v1",
      "manifest_id": "1794914457909949758",
      "source_name": "arxiv-ai-infra-inference-ops",
      "published_date": "2026-11-17"
    }
  ],
  "run_these_manifests": {
    "title": "Run these Manifests yourself",
    "summary": "Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.",
    "manifest_ids": [
      "1794914457909949767",
      "1794914457909949777",
      "1794914457909949760",
      "1794914457909949762",
      "1794914457909949773",
      "1794914457909949780",
      "1794914457909949758"
    ],
    "steps": [
      {
        "n": 1,
        "title": "Get a Synorb Key",
        "text": "Click Generate Free Agent Key at https://synorb.com/connect/start, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{\"client\":\"your-agent-name\"}`. The key is the `api_key` field of the answer. Save the key. Existing 72 Hour Agent Demo keys keep their issued limits. New keys require email OTP verification. An existing demo key lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons."
      },
      {
        "n": 2,
        "title": "Read each cited Manifest by id",
        "text": "REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief."
      },
      {
        "n": 3,
        "title": "Check the claim",
        "text": "Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them."
      }
    ],
    "mcp_url": "https://mcp.synorb.com/mcp",
    "connect_url": "https://synorb.com/connect/start",
    "request": {
      "method": "GET",
      "url_template": "https://api.synorb.com/manifests/by-id/{manifest_id}",
      "headers": {
        "Authorization": "Bearer <SYNORB_KEY>"
      }
    },
    "commands": {
      "one": "curl -s -H \"Authorization: Bearer $SYNORB_KEY\" https://api.synorb.com/manifests/by-id/1794914457909949767",
      "all": "for id in 1794914457909949767 1794914457909949777 1794914457909949760 1794914457909949762 1794914457909949773 1794914457909949780 1794914457909949758; do\n  curl -s -H \"Authorization: Bearer $SYNORB_KEY\" \"https://api.synorb.com/manifests/by-id/$id\"\ndone"
    }
  },
  "links": {
    "html": "https://hangingcontext.com/stories/2026-11-17/inference-energy-bill-comes-due/",
    "json": "https://hangingcontext.com/stories/2026-11-17/inference-energy-bill-comes-due/index.json",
    "markdown": "https://hangingcontext.com/stories/2026-11-17/inference-energy-bill-comes-due/index.md",
    "newsroom": "https://hangingcontext.com/n/artificial-intelligence/",
    "edition": "https://hangingcontext.com/editions/2026-11-17T12/"
  },
  "related": [
    {
      "url": "https://hangingcontext.com/stories/2026-11-22/where-your-prompts-actually-go/",
      "headline": "Three inference providers disclosed prompt failures in the same week",
      "slug": "where-your-prompts-actually-go"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-11-22/the-weeks-plumbing-papers/",
      "headline": "Three preprints fix the plumbing: which data to keep, which causes matter, when robots must stop",
      "slug": "the-weeks-plumbing-papers"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-11-22/the-agent-platform-standardizes/",
      "headline": "The agent platform is standardising: protocols, observability, governance",
      "slug": "the-agent-platform-standardizes"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-11-22/patch-the-ai-stack/",
      "headline": "Four patches, one lesson: the AI stack's weak points are the boring parts",
      "slug": "patch-the-ai-stack"
    }
  ],
  "sponsor": {
    "label": "From Synorb",
    "title": "Run the cited Manifests yourself",
    "text": "Every claim in this paper points to a Manifest id. Get a Synorb Key and read the Manifests behind it. Existing 72 Hour Agent Demo keys keep their issued limits. New keys require email OTP verification. An existing demo key lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons.",
    "cta": {
      "label": "Get a key",
      "url": "https://synorb.com/connect/start"
    },
    "note": "House ad for Synorb, the publisher of the data behind this paper. Not a news item."
  },
  "disclosure": "Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.",
  "reuse": {
    "version": "1",
    "summary": "Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product.",
    "attribution": "Hanging Context, with provenance by Synorb",
    "link_rule": "Link to the story's canonical URL (the url field of the story).",
    "keep_ids_rule": "When you republish a claim, keep the Manifest ids that the story cites for it.",
    "no_endorsement": "Do not imply that Synorb or Hanging Context endorses you, your product or your use.",
    "license_url": "https://hangingcontext.com/terms/reuse/",
    "machine_url": "https://hangingcontext.com/.well-known/reuse.json",
    "canonical_url": "https://hangingcontext.com/stories/2026-11-17/inference-energy-bill-comes-due/"
  }
}
