Long context gets cheaper to run, not just longer

Training longer without paying for the whole sequence

Google Cloud says its MaxText framework now supports long-context training up to 14 million tokens on Cloud TPUs, and version 0.2.4 introduces four context-parallel strategies: All-gather, A2A, P2P and USP. The Tokamax Splash Attention kernel powers the new context-parallel strategies. The headline number is the 14 million tokens, but the mechanism is the point: the work is split across devices so that no single device has to hold the whole sequence. m:…7903

The new strategies manage attention compute and memory without materialising full key-value sequences in high-bandwidth memory, according to Google. Developers can use MaxText version 0.2.4 to achieve high Model FLOPs Utilisation across large-scale model training workflows. In plain terms, the framework is trying to keep the hardware busy rather than waiting on memory, which is where long-context training usually loses its efficiency. A longer context is only useful if the training run does not collapse under its own memory bill, and this update is aimed squarely at that collapse. m:…7903

Longer context also needs a better test, and LongBanana is a preprint benchmark built for the image side of that problem. It is an expert-designed and expert-verified benchmark for long-context multi-reference image synthesis, and it contains 90 tasks organised by a 3-by-3 taxonomy. Each task includes about 32 reference images, 20 constraints and 50 evaluation items, yielding 4,670 expert-specified atomic evaluation items in total. The benchmark uses a graph-grounded evaluation design that converts node, edge and whole-graph obligations into checklist items, so a model is graded on whether it used the dispersed visual evidence and rejected the distractors, not just whether the final image looks plausible. m:…2284

Reuse the reasoning instead of re-solving

The cheapest token is the one you never generate, and CPU-ARC is built on that idea. Large language models conflate task-solving with the execution of the solutions, according to the preprint, which forces them to solve every problem from scratch. CPU-ARC is a method for constructing CPU-executable agentic reasoning caches from large language model reasoning traces. Once a task has been reasoned through, applying its solution to a new problem instance can be done efficiently and with little involvement from language models. m:…8854

The reported trade is striking. Agentic reasoning caches show on-par performance with frontier large language models across a range of difficult mathematical and scientific tasks, and are 16 to 291 times cheaper to invoke than the source model. The caches are also fully explainable, continually trainable and robust to input perturbations, according to the authors. That combination matters for a developer: a cached solution can be inspected and updated, where a fresh model call is a black box each time. The open question, stated in the preprint material itself, is how the system handles new problems that break the assumptions of the cached reasoning, which is exactly where a cache stops being a shortcut and starts being a risk. m:…8854

LongBanana's own results point the same way. Minimal agent harnesses improve over direct generation in long-horizon image generation tasks, and the LongBanana-Gen agent-harness framework, with a ReAct-Plan-Exec-Reflect workflow, yields harness-dependent gains by strengthening evidence grounding and repair. Expert labels for the benchmark achieved 89.4% exact agreement and 0.83 quadratic weighted kappa on a 15% double-labelled subset, which gives the benchmark itself a firmer footing than a typical crowd-labelled test. The lesson for developers is that the harness around the model, and the cache around the reasoning, may now matter as much as the model. m:…2284

Squeeze the cache that long context creates

Long context creates a memory problem at inference time, and LowRAM attacks it directly. LowRAM compresses key-value caches by predicting values from keys within each attention head and applying cross-layer low-rank key compression, enabling higher decoding throughput and composability with quantisation and eviction. Its value cache amortisation fits a lightweight per-head predictor at test time and stores residuals only for the tokens it reconstructs worst, giving 4 times near-lossless value compression. The idea is to treat keys and values as structurally distinct, rather than compressing them as if they were the same kind of data. m:…5407

The numbers, as reported in the preprint, are large. LowRAM achieves up to 5 times higher peak decode throughput at 4 times key-value compression, and it composes with quantisation and eviction to reach 26.9 times compression at 94.5% of baseline performance on long contexts. Predictors in LowRAM can be meta-learned to reduce prefill overhead, or fitted from scratch for one-off costs like multi-query inference. That last detail matters operationally: the method can be tuned for a repeated workload or fitted fresh for a one-off, rather than forcing one approach on both. m:…5407

LaProx, another preprint, takes a different angle on the same cache. It models multiplicative interactions between attention maps and projected value states to quantify token contributions, and it enables globally comparable importance scores for tokens, allowing model-wide selection instead of local, head-wise decisions. LaProx maintains model performance with only 5% of the key-value cache, and achieves up to 2 times accuracy loss reduction under extreme compression compared with state-of-the-art baselines. The shift from head-wise to model-wide selection is the conceptual move: instead of each attention head deciding on its own what to keep, the system compares tokens across the whole model. m:…5090