The GPU memory hierarchy is the new inference frontier: flash, KV cache, prefetching, direct storage

Four papers, one tier

Four preprints posted on October 23 attack the same layer of the stack: the memory tier between the GPU and disk. One extends flash lifetime for offloaded KV caches; one compresses KV caches for speculative decoding; one orchestrates GPU prefetching; one rewrites the direct path from GPU to storage. Four separate papers, one converging diagnosis: in long-context serving, the bottleneck has moved below the compute kernel, into where data rests and how it moves. m:…4590 m:…1418 m:…4300 m:…5484

Start with the flash tier. LLM serving routinely offloads KV caches to SSDs when GPU memory runs short, but the resulting write traffic burns through flash lifetimes — premature SSD failure is the failure mode Gandalf targets. The paper proposes a lifetime-aware admission policy that distinguishes attention types and prioritizes admitting full-attention KV caches, which the authors' measurements show cuts SSD writes without reducing cache hit rate. The reported headline: SSD lifetime extends 7.3X on average in LLM serving environments. That is a preprint measurement, not a vendor guarantee, but the framing is the point — flash wear, not flash speed, is becoming the constraint to engineer around. m:…4590

RIFT works one tier up, on the draft model's own cache. Speculative decoding depends on small draft models whose candidate tokens the main model checks — but compressing the drafter's KV cache degrades its accuracy during long-output generation, a failure the authors call decode staleness. Their answer combines an adaptive refresh strategy, which dynamically balances prefill information with recent tokens, with drafter training that uses sparse attention masks to make the model more compressible. The measured result: more than 98% of full-cache acceptance length on 32K-token inputs and 4K-token outputs, inside an 8K-token KV cache budget. Same tier, different trade — cache size down, drafter accuracy preserved. m:…1418

SnakeCharmer moves closer to the silicon, into GPU prefetching. Most prefetchers couple address prediction to the act of prefetching; SnakeCharmer decouples the two with a backend control framework that jointly manages the utility, timeliness, and placement of prefetched data — what to fetch, when, and where to put it. The authors report 12.8% better performance and 20.8% better energy efficiency than the Snake baseline on A100 GPUs. The architectural claim matters more than the number: the backend-control principles are compatible with multiple frontend predictors, including APOGEE, so this is a control framework rather than another predictor to swap in. m:…4300

The last piece is the road to disk itself. NVIDIA's GPU Direct Storage lets GPUs talk to NVMe without CPU copies, but its kernel-traversal overhead becomes the bottleneck for storage-bound inference. uGDS moves the NVMe control path into user space with workload-aware scheduling, and the reported numbers are striking: up to 12.12x batch-read and 30.69x batch-write bandwidth compared to standard GDS, 4-KiB synchronous write latency down 84.1% with two threads, and in LMCache, median time-to-first-token cut by up to 62.7% with serving goodput up 2.63x. The kit also supports both NVIDIA CUDA and AMD HIP, which matters as inference fleets stop being single-vendor. m:…5484

Why this tier is the frontier

The throughline across all four papers: GPU memory capacity is growing slower than the context windows and batch sizes developers want to serve. Offload to SSD, compress the cache, predict and place prefetches, stream straight from storage — the serving stack's memory tier is becoming a programmable system instead of the hardware's problem. Four papers in one week, four different rungs of the hierarchy, one converging idea. m:…4590 m:…4300 m:…5484

For AI developers, the practical read is that the storage path is becoming a tuning surface rather than a fixed cost. Gandalf's admission policy is a design you could imagine inside any cache manager; uGDS is a concrete user-space kit with published code. The question worth bringing to your infrastructure team: which layer of your memory hierarchy is currently treated as the hardware's problem, and what happens to your serving margins if you stop treating it that way? This suggests, rather than proves, that the next round of inference cost wins comes from data movement, not from faster matrix math. m:…4590 m:…5484

The honest caveats

Every number in this story rests on the authors' own preprint measurements — none of it is peer-reviewed or independently reproduced. The gains were reported on specific setups: A100 GPUs, LMCache, particular workload shapes. A result like 7.3X SSD lifetime extension is exactly the kind that needs real cluster telemetry before it becomes a buying or provisioning decision. Treat these as four directions, not four benchmarks. m:…4590 m:…4300 m:…5484