The energy bill hides in decode: power caps and speculative tricks can cost more than they save

Speculation only saves energy when tokens are accepted

In a preprint, researchers account for the energy of speculative decoding, the technique in which a small draft model proposes tokens and a large model verifies them, and find that the savings depend on details that headline benchmarks rarely report. Energy savings are highly variable and depend on token acceptance rates across models and text types, and verification, not drafting, is the dominant energy cost on H100 and H200 GPUs. At large batch sizes, the authors report, speculative decoding can actually increase energy per token compared with standard decoding, because the workload shifts into a compute-bound regime. The decision that follows is practical: an operator should enable speculation where acceptance is high and batches are small, and measure energy per accepted token, not per generated token, because a rejected draft token is work paid for and thrown away. m:…1225

The same research group, in a second preprint, finds a similar gap between the control and the outcome for GPU power capping. Power capping is ineffective for LLM decode on NVIDIA H200 GPUs, the authors report, because decode kernels are memory-bound and the GPU idles between kernel launches, so capping power does not remove work that was never being done. Locking the streaming multiprocessor clock instead is reported to recover up to 32% of decode energy without significant throughput loss. The point generalises: a control that works for compute-bound training may do nothing for memory-bound decode, and the phase of inference, prefill or decode, decides which control helps. m:…8925

The authors also note that recurrent architectures, naming Gated DeltaNet and Mamba2, avoid prefill penalties compared with quadratic attention on long prompts. That is a further reason to measure by phase and by architecture rather than by a single efficiency number: a model that looks expensive in one phase may be cheap in the phase that dominates a given workload, and the reverse. m:…8925

Where the work runs matters as much as how it runs

A third preprint moves the question from the GPU to the map. Aceso is a system for placing microservices across regions in geographically constrained settings, where the usual assumption of a global cloud footprint does not hold. Using insight-based search space pruning to keep the optimisation tractable, it balances carbon, cost and latency, and the authors report reductions of up to 37.4% in carbon emissions and up to 3.36% in operating costs compared with static single-country deployments, while maintaining latency targets. The paper is aimed in particular at smaller operators and latency-sensitive applications, the deployments least able to absorb a carbon or cost penalty for the sake of a simple placement rule. m:…2682

Read together, these preprints argue for an energy accounting habit rather than any single trick. Report energy per accepted token for speculative decoding, report decode and prefill separately for power controls, and report carbon and cost together for placement, with latency held as a constraint rather than traded away silently. An operator that buys an efficiency feature on its headline number, without checking the phase, the batch size and the acceptance rate, may be paying more for energy while believing the bill went down. m:…1225 m:…8925 m:…2682