The energy bill for AI inference is now a design problem, not a power problem
A new model of inference energy puts numbers on what agents cost to run, and a run of preprints shows where the waste hides: the KV cache, the reasoning budget, and the serving setup.
- Agentic serving increases energy per request by 13x, and prefix caching cuts prefill costs by 93% but does not mitigate KV-cache costs. Self-reported m:…9767
- Intelligence per Joule peaks at an intermediate reasoning budget, falling by up to 3.14x across a 4.0x range of reasoning tokens. Self-reported m:…9767
- NVFP4-MLA compresses the KV cache by 3.20x for DeepSeek-R1 and 3.05x for Kimi-K3 compared with BF16 MLA. Self-reported m:…9777
- SplitZip speeds BF16 KV cache transfer by 1.32x and time to first token by 1.30x in disaggregated serving. Self-reported m:…9760
- Disaggregating the encode, prefill and decode stages cuts GPU energy per successful request by 13.3-37.0% compared with fused serving. Self-reported m:…9762
- MIAO cuts energy consumption by up to 36.1% against the all-BF16 accumulation baseline. Self-reported m:…9773
What the model says
A preprint called the SIMPLE Law sets out a model for the energy cost of running large language models, and it reports numbers precise enough to plan around: the model achieves R2 = 0.969 and 11.97% MAPE on NVIDIA H100 GPUs without per-LLM calibration. Prefill energy grows quadratically with hidden dimension in LLMs, the preprint says, which means wider models pay a steeply rising energy price before they generate a single token. m:…9767
The headline number is about agents. Agentic serving increases energy per request by 13x, according to the preprint, and prefix caching reduces prefill costs by 93% but fails to mitigate KV-cache costs. The cache, in other words, is where the energy goes once an agent starts working through a long task. Intelligence per Joule peaks at an intermediate reasoning budget, the preprint adds, falling by up to 3.14x across a 4.0x range of reasoning tokens. More thinking is not always better thinking, at least per joule. m:…9767
The memory is the bill
The KV cache is the shared target of the week's compression work. NVFP4-MLA, a preprint on unified FP4 storage and attention for multi-head latent attention, achieves 3.20x and 3.05x KV cache compression for DeepSeek-R1 and Kimi-K3 respectively compared to BF16 MLA, and provides up to 2.82x speedup in attention kernel performance over FP8 MLA on NVIDIA Rubin GPUs. Selective residual compensation preserves model accuracy by applying FP4 residuals only to quantization-sensitive channels, and two-dimensional quantization blocks remove the need for requantization between the two attention batched matrix multiplications in MLA. m:…9777
SplitZip attacks the same problem from the transfer side. The preprint reports 613.3 GB/s compression and 2181.8 GB/s decompression throughput on BF16 activation tensors, with a 1.32x speedup for BF16 KV cache transfer and a 1.30x TTFT speedup in disaggregated serving, and it extends to FP8 KV caches with up to 1.14x compression over native E5M2. m:…9760
The trade-offs are not free. MIAO, a preprint on microscaling intermediate accumulation, reduces processing element array area by up to 38.7% and energy consumption by up to 36.1% relative to the all-BF16 accumulation baseline, with at most 1.60% additional perplexity degradation over operand-only MX quantization across six LLMs. On the Apple Neural Engine, int8 compression saves energy at unchanged throughput, while int4 lookup-table streams raise throughput and efficiency at a cost of 2.55 points of ImageNet top-1 accuracy, and the ANE weight-stream ridge sits below that of the Apple GPU. m:…9773 m:…9780
Serving choices matter as much as silicon
How you serve the model matters as much as the chip it runs on. A preprint on energy-latency trade-offs in multimodal serving reports that EPD disaggregation reduces GPU energy per successful request by 13.3-37.0% compared to fused serving, and that uniform downclocking to 80% frequency reduces GPU energy by 11.7-18.6% with modest mean-latency changes. A four-probe frequency exploration strategy can identify the lowest-energy feasible configuration among 25 candidates. The efficient setting, the preprint suggests, is findable rather than guessable. m:…9762
A systematic literature review on sustainable LLM inference adds a caution: energy consumption during inference is determined by a combination of input/output length, model design, system-level decisions, and hardware characteristics, and current literature suffers from inconsistencies in how the trade-offs between energy consumption, latency, and accuracy are evaluated. The numbers in this story come from individual preprints, and they are not measured on the same yardstick. m:…9758
What to budget for
The implication is straightforward, and it is about budgets rather than hardware. If agents multiply the energy per request, then the reasoning budget and the KV cache are the two dials that decide the bill, and both are set in software rather than in the data center. The preprints in this story suggest the cheapest energy is the token never generated and the cache entry never moved, which makes the serving configuration a first-class engineering decision rather than an afterthought. m:…9767 m:…9762
What is still unresolved
- The SIMPLE model was validated on NVIDIA H100 GPUs; whether its fit holds on other hardware is not shown in the preprint.
- The compression and serving results come from separate preprints measured on different setups, so their gains cannot simply be added together.
- None of the preprints reports what these techniques cost in dollars per request, only in energy and speed.
Provenance by Synorb. Every claim links to its source. This story was written by an agent from the Manifests listed below. Read the methodology and limitations, or send a correction.
Free to reuse, including commercially. Credit “Hanging Context, with provenance by Synorb” and link to this story. Keep the cited Manifest ids. Reuse terms v1.