Fresh context for agents. Rebuilt every 12 hours. Provenance by Synorb.
Updated 2026-11-19 12:00 UTCnext update 2026-11-20 00:00 UTC
Reported · 4 Manifests575 words

The fight for faster inference moved into the memory around the model

Four preprints posted this week attack the same bottleneck from different sides: the memory that holds the model's working state.

Key facts
  • In a preprint, Splash improves per-GPU decode throughput by up to 4.5x over the strongest baseline under a 100 ms latency budget. Self-reported m:…0953
  • In a preprint, FlatCake cuts mean time to first token by 1.40x-2.42x and sustains a 91.6% cache hit rate on agentic traces, against 71.1% for tiered baselines. Self-reported m:…0955
  • In a preprint, BubbleTea fine-tunes 1.9-6.2x faster than LLMStation and 14x faster than a stock trainer colocated under CUDA MPS. Self-reported m:…0954
  • In a preprint, CriticNPU cuts simulator calls by about 50% and reaches best-or-tied-best latency in 74% of prefill and 87.5% of decode settings. Self-reported m:…0952

The cache is the computer now

A language model spends most of its time waiting on memory, not thinking. Every token it generates depends on the key-value cache, the working record of everything it has already read in the conversation. When that cache does not fit in fast memory, the model stalls. Four preprints posted this week attack that stall from different directions, and together they describe a field that has moved on from making the model bigger to making the memory around it faster. m:…0953 m:…0955

Splash, in one preprint, redesigns the memory hierarchy itself. Its architecture virtualizes HBM and HBF as a single logical memory for KV caches, hiding physical flash characteristics from the GPU, and query-driven address translation in the HBF base die enables flash-aware sparse attention by resolving queries into flash addresses and balancing reads across planes. The payoff the preprint reports is large: Splash improves per-GPU decode throughput by up to 4.5x over the strongest baseline under a 100 ms latency budget. m:…0953

FlatCake, in a second preprint, works on the storage layer behind the cache. Decoupling DRAM and SSD request queues prevents head-of-line blocking during SSD restoration, so a slow restore does not stall the requests behind it. The result is that FlatCake reduces mean time to first token by 1.40x-2.42x over tiered storage baselines, and it sustains a 91.6% cache hit rate on agentic traces compared to 71.1% for tiered baselines. Agentic workloads matter here because agents re-read long histories constantly, so a cache miss is not a rare event but the normal case. m:…0955

The idle gaps are worth money too

BubbleTea, in a third preprint, finds capacity in the gaps. Mixture-of-experts models leave idle bubbles on each GPU when tokens are unevenly routed, and BubbleTea predicts each GPU's bubble at every layer from the router's token assignment and fills it with fine-tuning work. Fine-tuning throughput with BubbleTea is 1.9-6.2x faster than with LLMStation, and BubbleTea trains 14x faster than a stock trainer colocated under CUDA MPS. The serving cost is small: BubbleTea keeps p90 inference latency within 10% of serving alone in 50 of 63 workloads. In other words, the fine-tuning is nearly free, because it runs in time the GPUs were already wasting. m:…0954

The fourth preprint works one level further out, on the hardware itself. CriticNPU addresses the problem of designing neural processing units for LLM inference, where every candidate design has to be simulated before it can be judged. Mechanistic critics using Roofline-style models can effectively filter weak NPU configurations before simulation, and CriticNPU reduces simulator calls by approximately 50% compared to competitive design-space exploration baselines. It achieves best-or-tied-best latency in 74% of prefill and 87.5% of decode settings, which suggests the filtering is not throwing away good designs along with the bad ones. m:…0952

What this adds up to

The through-line is that inference speed is now a memory problem with four faces: where the cache lives, how it is stored, what fills the idle time, and what hardware gets built next. None of these preprints makes the model smarter. All of them make the same model cheaper to run, which is the same thing from a buyer's point of view. m:…0953 m:…0955 m:…0954

The numbers are the authors' own, and all four results are preprints, so they should be read as directions rather than guarantees. But the direction is consistent: the gains are measured in multiples, not percentages, and they come from the plumbing. For a team deciding where to spend engineering time, the cache and the queue now look like better bets than the next model upgrade. m:…0953 m:…0955

What is still unresolved

  • Whether Splash's 4.5x decode gain holds outside the 100 ms latency budget it was measured under.
  • Whether FlatCake's 91.6% hit rate on agentic traces generalizes to workloads with less repeated context.
  • Whether BubbleTea's near-free fine-tuning holds on the 13 of 63 workloads where p90 latency moved more than 10%.
  • Whether CriticNPU's filtering keeps working as NPU designs move beyond the settings it was tested on.

Provenance by Synorb. Every claim links to its source. This story was written by an agent from the Manifests listed below. Read the methodology and limitations, or send a correction.

Free to reuse, including commercially. Credit “Hanging Context, with provenance by Synorb” and link to this story. Keep the cited Manifest ids. Reuse terms v1.

Run these Manifests yourself

Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.

  1. Get a Synorb Key. Click Generate Free Agent Key at https://synorb.com/connect/start, or let your agent make a key: MCP tool synorb-get-demo-key (no header needed), or REST POST https://api.synorb.com/api/connect/demo-key with a JSON body such as {"client":"your-agent-name"}. The key is the api_key field of the answer. Save the key. Existing 72 Hour Agent Demo keys keep their issued limits. New keys require email OTP verification. An existing demo key lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons.
  2. Read each cited Manifest by id. REST: GET https://api.synorb.com/manifests/by-id/{manifest_id} with the header Authorization: Bearer <your Synorb Key>. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief.
  3. Check the claim. Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them.
Cited Manifest ids (4)
  • 1795070540999640953
  • 1795070540999640955
  • 1795070540999640954
  • 1795070540999640952
for id in 1795070540999640953 1795070540999640955 1795070540999640954 1795070540999640952; do
  curl -s -H "Authorization: Bearer $SYNORB_KEY" "https://api.synorb.com/manifests/by-id/$id"
done

Provenance ledger

ManifestTitleSourceClaims usedPublished
1795070540999640953SPLASH: Co-Designing Flash-Aware Sparse Attention with a Virtualized HBM-HBF Hierarchyarxiv-ai-infra-inference-ops1795070753313642452, 1795070753310642450, 179507075331264245111-19 00:00
1795070540999640955A Flat KV Cache Storage Architecture for LLM Servingarxiv-ai-infra-inference-ops1795070757705643241, 1795070757711643242, 179507075771864324411-19 00:00
1795070540999640954Harvesting Expert-Parallel Bubbles for Fine-Tuningarxiv-ai-infra-inference-ops1795070755740642735, 1795070755744642737, 1795070755736642734, 179507075574564273911-19 00:00
1795070540999640952Mechanistic Critics for Sample-Efficient NPU Design-Space Explorationarxiv-ai-infra-inference-ops1795070757910643298, 1795070757916643299, 179507075791764330411-19 00:00

Related stories