---
title: The serving stack's memory wars: unified KV addressing hits 2.51x, GPU-NPU disaggregation cuts cost 1.72x
description: Five new preprints attack LLM inference cost from the same direction: move the KV cache smarter. UlaCache reports 2.51x peak decode throughput on B300, HAWK offloads decode to NPUs at 1.72x the cost efficiency, and KVLearn trims time-to-first-token 56% by learning what to keep.
canonical: https://hangingcontext.com/stories/2026-10-22/serving-stack-memory-wars/
last-updated: 2026-10-22
url: https://hangingcontext.com/stories/2026-10-22/serving-stack-memory-wars/
published: 2026-10-22T12:45:39Z
edition: 2026-10-22T12
newsroom: artificial-intelligence
beat: infrastructure
trust: reported
source_count: 5
cited_manifest_ids: ['1792665412512696698', '1792665412512616718', '1792665412512305234', '1792665412512174125', '1792665412513743118']
---

# The serving stack's memory wars: unified KV addressing hits 2.51x, GPU-NPU disaggregation cuts cost 1.72x

*Five new preprints attack LLM inference cost from the same direction: move the KV cache smarter. UlaCache reports 2.51x peak decode throughput on B300, HAWK offloads decode to NPUs at 1.72x the cost efficiency, and KVLearn trims time-to-first-token 56% by learning what to keep.*

Reported · 5 sources · Artificial Intelligence · Published 2026-10-22 12:45 UTC

## Key facts

- UlaCache's unified logical addressing for KV cache management, with parallel bucketed LRU eviction, reports 2.51x peak decoding throughput on B300 hardware in tensor-parallel mode versus state-of-the-art baselines. (self-reported) [cites: 1792665412512696698]
- HAWK's heterogeneous GPU-NPU phase-disaggregated inference reports 1.72x higher cost efficiency and 1.26x higher power efficiency than GPU-only systems, while staying within 7.88% of GPU-only performance under identical latency constraints. (self-reported) [cites: 1792665412512616718]
- KVLearn's learning-based cache retention reports up to 56% lower time-to-first-token than no caching, 33% lower than Mooncake-style baselines, and 53% less inter-node KV transfer volume than LRU pooling. (self-reported) [cites: 1792665412512305234]
- Dooly's redundancy-aware profiling reports 56.4% fewer profiling GPU-hours than standard approaches, with simulation error of 5% MAPE on time-to-first-token and 8% on time-per-output-token. (self-reported) [cites: 1792665412512174125]
- SubTernary's CPU speculative decoding reports 1.35x speedup over llama.cpp's speculative decoding and 3.99x over autoregressive decoding; systems runtime changes drove 35.0% of the gain versus 6.0% from the ternary SIMD kernel. (self-reported) [cites: 1792665412513743118]

## One address space for the KV cache

The KV cache is the tax every long-context serving system pays: the longer the context, the more memory each request pins down, and the faster the GPUs fill up. In a preprint, a preprint proposes UlaCache, which gives the KV cache a unified logical address space. The inference engine manages cache blocks across GPU and CPU pools without caring where they physically live, and a parallel bucketed LRU policy, PB-LRU, manages the shared GPU pool by updating recency metadata and selecting eviction victims in parallel. The reported peak is 2.51x decoding throughput on B300 hardware in tensor-parallel mode compared with state-of-the-art baselines. [cites: 1792665412512696698]

The implication, if that number holds up, is that memory management overhead rather than raw compute is the binding constraint on decode throughput, and that an addressing abstraction can buy back headroom without new hardware. That is the kind of claim worth testing on your own cluster before believing, but it is also the cheapest kind of claim to test: it is software. [cites: 1792665412512696698]

## Decode doesn't need a GPU

HAWK starts from a different observation: GPUs sit underutilized during the decode phase, so why not hand decode to cheaper NPUs? The catch has always been moving the KV cache between devices with different tensor layouts. HAWK's answer is tensor-layout-aware RDMA buffers that let the NPU consume transferred KV cache tensors directly, without an extra data-rearrangement pass. In the authors' evaluation the heterogeneous setup reached 1.72x the cost efficiency and 1.26x the power efficiency of a GPU-only system, while holding inference performance within 7.88% of the GPU-only baseline under identical latency constraints. [cites: 1792665412512616718]

> **From Synorb** — Run the cited Manifests yourself. Every claim in this paper points to a Manifest id. Get a Synorb Key and read the Manifests behind it. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; 3 of them are starter Beacons. [Get a key](https://synorb.com/connect/start)


That 7.88% figure does the important work in the paper. It converts the disaggregation pitch from a latency gamble into a cost question: if serving quality stays within single-digit percent of the baseline while cost efficiency nearly doubles, the GPU-only default gets harder to defend, at least for workloads whose latency budgets tolerate the transfer. As with all five of these papers, the figures are the authors' own measurements. [cites: 1792665412512616718]

## Learn what to keep

KVLearn works the same disaggregated setup, with prefill and decode on separate nodes, but asks a narrower question: once the KV cache is scattered across the cluster, what should you keep? The framework learns the answer. A Prefix Reuse Predictor estimates how likely a cached prefix is to be reused, from structural and temporal features of the prefix, without touching model weights; a cost-aware retention score and an adaptive threshold controller then adjust admission online as memory pressure and hit rates change. The reported results are up to 56% lower time-to-first-token than no caching, 33% lower than Mooncake-style baselines, and 53% less inter-node KV transfer volume than LRU pooling. [cites: 1792665412512305234]

## Profile cheaper, run on CPU

Two more papers work the edges of the same cost problem. Dooly attacks profiling, the simulation runs teams do before choosing a serving configuration. It exploits structural redundancy across configurations: one inference pass plus taint propagation to label input dimensions lets it skip re-profiling operations already in its latency database. The authors report 56.4% fewer profiling GPU-hours than existing approaches, with simulation error within 5% MAPE on time-to-first-token and 8% on time-per-output-token. [cites: 1792665412512174125]

SubTernary goes the other direction and makes the CPU a viable inference target. Speculative decoding on CPUs is bottlenecked by context-swapping latency and memory allocation overhead, and SubTernary answers with unified graph caching and transactional KV-cache isolation. The reported speedups are 1.35x over llama.cpp's speculative decoding and 3.99x over standard autoregressive decoding. The ablation is the interesting part: systems runtime optimizations drove 35.0% of the throughput gains, while a 256-bit AVX2 ternary SIMD kernel added only 6.0%. The plumbing beat the kernel. [cites: 1792665412513743118]

## What is still unresolved

- UlaCache, HAWK and SubTernary mention no code or dataset release, so the headline numbers cannot be independently reproduced from the available evidence.
- HAWK's 7.88% performance figure holds under the authors' latency constraints; other workload mixes may differ.
- All five papers are preprints and their results rest on the authors' own evaluations.

## Decision card

- **Useful for:** Teams running long-context or high-throughput LLM serving clusters; Operators evaluating heterogeneous GPU-NPU hardware
- **Not for:** Readers looking for peer-reviewed results; all five papers are preprints
- **Compare against:** Standard GPU-only serving stacks and static KV cache eviction policies

## Provenance ledger

| Manifest id | Title | Source | Claims used | Published |
|---|---|---|---|---|
| `1792665412512696698` | UlaCache: Unified Logical Addressing for Sparse LLM Serving | [arxiv-ai-infra-inference-ops](https://arxiv.org/pdf/2610.29820v1) | 1792665978994361581, 1792665978998507839, 1792665978996200187 | 10-22 00:00 |
| `1792665412512616718` | HAWK: Heterogeneity-Aware KV Cache Transfer for Efficient GPU-NPU Phase-Disaggregated LLM Inference | [arxiv-ai-infra-inference-ops](https://arxiv.org/pdf/2610.29873v1) | 1792665980970169254, 1792665980973973933, 1792665980974708202 | 10-22 00:00 |
| `1792665412512305234` | To Keep or Not to Keep: Learning KV Cache Retention in Disaggregated LLM Serving Systems | [arxiv-ai-infra-inference-ops](https://arxiv.org/pdf/2610.28637v1) | 1792665977657148873, 1792665977661525710, 1792665977665975733 | 10-22 00:00 |
| `1792665412512174125` | Dooly: Configuration-Agnostic, Redundancy-Aware Profiling for LLM Inference Simulation | [arxiv-ai-infra-inference-ops](https://arxiv.org/pdf/2605.07985v3) | 1792665983052391276, 1792665983055598382, 1792665983056726375 | 10-22 00:00 |
| `1792665412513743118` | Accelerating CPU Speculative Decoding via Unified Graph Execution and Transactional KV Isolation | [arxiv-ai-infra-inference-ops](https://arxiv.org/pdf/2610.28671v1) | 1792665980855975943, 1792665980857184903, 1792665980852778335 | 10-22 00:00 |

## Run these Manifests yourself

Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.

1. **Get a Synorb Key.** Click Generate Free Agent Key at https://synorb.com/connect/start, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{"client":"your-agent-name"}`. The key is the `api_key` field of the answer. Save the key. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; 3 of them are starter Beacons.
2. **Read each cited Manifest by id.** REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief.
3. **Check the claim.** Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them.

Cited Manifest ids:

- `1792665412512696698`
- `1792665412512616718`
- `1792665412512305234`
- `1792665412512174125`
- `1792665412513743118`

```sh
for id in 1792665412512696698 1792665412512616718 1792665412512305234 1792665412512174125 1792665412513743118; do
  curl -s -H "Authorization: Bearer $SYNORB_KEY" "https://api.synorb.com/manifests/by-id/$id"
done
```

> **From Synorb** — Connect your agent over MCP. Add https://mcp.synorb.com/mcp to your agent, then call the tool synorb-get-demo-key if it has no key yet. It needs no email and no form. Keys are free for 72 hours, and new keys are limited per network. [How to connect](https://synorb.com/connect/start)

## Related stories

- [Scaling laws get sharper: MoE capacity theory, VLM transfer prediction, and a 20k-parameter jet tagger](https://hangingcontext.com/stories/2026-10-22/scaling-laws-three-ways/)
- [Robot navigation's pretraining playbook: train smarter, transfer further, simulate once](https://hangingcontext.com/stories/2026-10-22/robot-navigation-pretraining-playbook/)
- [Reasoning RL gets surgical: prefix-level trust regions, retry-aware credit, and risk-sensitive GRPO for diffusion models](https://hangingcontext.com/stories/2026-10-22/reasoning-rl-surgical/)
- [Reasoning leaves the token stream: four preprints on non-autoregressive intelligence](https://hangingcontext.com/stories/2026-10-22/reasoning-beyond-autoregression/)

## Links

- HTML: https://hangingcontext.com/stories/2026-10-22/serving-stack-memory-wars/
- JSON: https://hangingcontext.com/stories/2026-10-22/serving-stack-memory-wars/index.json
- Markdown: https://hangingcontext.com/stories/2026-10-22/serving-stack-memory-wars/index.md

Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.

---
Reuse (terms v1): Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product. Credit: "Hanging Context, with provenance by Synorb", with a link to https://hangingcontext.com/stories/2026-10-22/serving-stack-memory-wars/. Terms: https://hangingcontext.com/terms/reuse/