Long context is not the bottleneck. Attention and cost are

What actually breaks in long inputs

Language model failure on long inputs in retrieval tasks stems from attention competition rather than sequence length, according to a paper on context rot. Neutral filler text up to 32,000 tokens had minimal impact on retrieval accuracy. But evidence-format statements placed in the same positions dropped accuracy to between 0.16 and 0.33. Length was not the problem; rival content competing for the model's attention was. m:…7821

The effect sizes are large. A single statement providing a rival value to the queried key multiplied loss by 9 to 110 times. Attention mass left on the evidence explained up to 58% of the variance in model loss, while log length explained at most 8%. Masking cost 45 accuracy points at 32,000 tokens in retrieval tasks. The study ran 126,473 trials, and the authors note it is correlational rather than a direct intervention on attention, so the mechanism is inferred, not proven. m:…7821

A second paper reaches a compatible conclusion from the engineering side: frequent context compaction preserves task accuracy across long-horizon and long-input tasks. Frontier language models sacrifice little to no task accuracy when maintaining short contexts through frequent compaction, and performance does not collapse on long-horizon software engineering tasks when the model takes at most one action per context. If the context can be compacted without losing accuracy, the long window is a convenience, not a requirement. m:…9460

Later tokens cost more, so stop sooner

The cost argument sharpens the point. Generating later tokens is costlier than generating earlier ones, and on frontier open-weight models the disparity between later and earlier tokens can rise to as much as 7 times. Every unnecessary step in a long agent run is therefore charged at the most expensive rate. The authors propose semantic state space models as a more efficient alternative for long-horizon work, a design that keeps a compact state rather than an ever-growing transcript. m:…9460

LoCo attacks the same cost from the training side. It uses local preference learning to teach reasoning models when to stop deliberating, detecting candidate overthinking states from the model's own traces, and its training data comes entirely from self-generated mathematical traces. Early stopping in LoCo mainly reduces redundant verification while preserving the verification the model still needs, according to the paper. m:…6306

The reported savings are substantial: completion length down 41.7% to 45.4% on mathematics benchmarks and 23.4% to 36.3% on code benchmarks, with effective transfer to code generation despite math-only training. A model that stops deliberating sooner is a model that bills fewer of those expensive late tokens, and LoCo's 7-billion-parameter model is publicly released for teams that want to test the claim themselves. m:…6306

Architectures that loop instead of lengthen

NVIDIA's Dory1.0-4B offers a structural version of the same idea. It is a 4-billion-parameter recurrent language model with elastic depth: increasing the loop count improves accuracy, with the largest gains on coding, agentic, and long-context benchmarks. With two or more loops, it outperforms Qwen3.5-9B and Gemma 4 variants on average across 12 benchmarks, despite being the smaller model. m:…4687

With four loops, Dory generates 1.3 to 2.5 times fewer tokens than with one loop, and because it caches keys and values only from the last loop, its KV-cache footprint stays equal to a non-looped model. It was pre-trained on 6 trillion tokens and supports a 256,000-token context, with quantized checkpoints released in BF16, MXFP8, NVFP4, and GGUF formats. The design trades repeated computation for memory, which is the opposite direction from simply buying a longer window. m:…4687

The practical advice that falls out of all three papers is consistent: protect the evidence from competition, compact context aggressively, and stop deliberating when the answer is already determined. The masked-filler protocol in the retrieval paper lost 11 to 18 points on MMLU at 30,000 tokens, a reminder that compaction itself has a cost if done carelessly. But the expensive default, keeping everything forever, looks worse with every new measurement. m:…7821