The edge stops waiting for the datacenter: four papers redraw the inference map

The attention pipeline moves into storage

On-device retrieval-augmented generation has a prefill problem: every query drags documents through PCIe and DRAM before the model ever touches them. A preprint posted this week, MemLLM, moves the attention pipeline directly into the storage layer, using per-channel accelerators and a DRAM-less dataflow so the traffic between storage and compute stays flat as the workload grows. m:…6682

The authors report an average end-to-end speedup of 4.06x over a GPU baseline for on-device RAG. A fused recomputation-query kernel halves the NAND read volume and removes write-back penalties, they report — the attention work happens where the data already lives. m:…6682

The shift that matters is architectural, not incremental. Inference stops being a thing that happens after data leaves storage, and the storage controller becomes a compute tier. If the numbers generalize beyond the authors' rig, the edge hardware roadmap stops being just GPUs and NPUs. m:…6682

The NPU hype meets the decode phase

For years the story was simple: the mobile NPU accelerates on-device LLM inference. A second preprint, PowerBench, runs a cross-layer analysis of mobile inference and finds the story breaks at the decode step. NPUs excel at compute-bound prefilling, the authors report, but CPUs outperform every other backend in memory-bound decoding. m:…2657

The performance gaps are not just silicon — they are software. Framework-induced differences in offloading strategies and quantization produce up to 10x swings on the NPU, the authors report. And the cheapest fix they found was not a new chip: tuning CPU frequency alone cuts energy by up to 39.7%, while an energy-oriented configuration on the NPU backend cuts energy up to 53.2%. m:…2657

For anyone buying edge hardware, the implication is that the NPU is a coprocessor for one phase of inference, not a replacement strategy — and the software stack around it looks like the bigger variable. m:…2657

A 46 MB model that holds LAMBADA

Meanwhile the model itself is shrinking. A third preprint argues that embedding tables — often the largest component of small language models — are overpriced. The authors replace trainable embedding tables with a frozen 16 MB vocabulary built from PCA and sign binarization, and report no performance degradation on standard benchmarks. m:…3203

Paired with a ternary BitNet b1.58 body, that frozen vocabulary yields a 46 MB model that holds performance on LAMBADA and PIQA — running at 615 to 740 tokens per second on laptop CPUs, a 5 to 6x speedup over 8-bit GGUF models. m:…3203

The frozen tables also survive what learned tables do not: vocabulary expansion. Learned tables collapse by 43% when new words are added; the frozen ones keep their rare-word robustness. That is a maintenance argument as much as a performance one — small models that can be updated without retraining their embeddings. m:…3203

Encoding the bandwidth, not the weights

The fourth paper, TRACE, addresses the other edge reality: devices rarely infer alone. In distributed edge inference, exchanging activations between devices dominates cost. TRACE replaces activation exchange with codebook-indexed representations, letting receivers do direct key/value lookups instead of redundant downstream computation. m:…0438

The paper reports up to a 22,624x reduction in communication overhead versus uncompressed sequence-parallel inference. Net effect: up to 1.27x end-to-end speedup and 1.89% higher accuracy than the current state-of-the-art vector-quantization-based distributed frameworks, with hierarchical clustering that adapts the communication bitrate at runtime without retraining. m:…0438

Taken together, the four papers point at one architecture: inference as a distributed, storage-close, CPU-tolerant workload. The datacenter remains where models are trained. It is increasingly not where they run. m:…0438