Agent memory is overbuilt: plain RAG over raw history beats the fancy systems
A preprint finds naive RAG over raw conversation history outperforms complex agentic memory in six of seven settings, while a parametric memory slashes retrieval overhead and a repair trick salvages bad memories.
- In a preprint posted this week, Raw-RAG — dense retrieval plus reranking over raw history feeding a reader LLM — outperformed complex agentic memory systems in six of seven benchmark-model settings, by 0.13 to 5.53 points, across LoCoMo, LongMemEval, MEME, and MemoryAgentBench. Self-reported m:…5077
- The authors' takeaway: preserving raw information matters more than elaborate memory transformations — transformed memory stores leave an accuracy gap relative to raw evidence, though raw-history retrieval's higher coverage of necessary evidence offsets its lower utilization. Self-reported m:…5077
- In a second preprint, Deep Fusion Memory integrates predicted parametric memory into intermediate hidden states via gated cross-attention, trained end to end with the backbone frozen — needing 32x fewer retrieval queries and 2048x fewer retrieved items than existing parametric memory approaches, and beating the strongest baseline by 12.71% on Mistral-7B-v0.3 and 11.19% on Qwen3-8B-Base. Self-reported m:…8543
- In a third preprint, dependency-guided rollback repair for memory-augmented agents achieves 85.3% recovery on a 150-case benchmark, while selective replay cuts LLM calls 41.8% versus full recomputation — at the cost of higher recurrence (26.6%) than LLM-judge repair (12.1%). Self-reported m:…8196
The embarrassment of simplicity
Agent-memory vendors sell elaborate machinery: memory that mutates, organizes itself into structures, and learns retrieval strategies. In a preprint posted this week, the authors test that machinery against the dumbest possible baseline — dense retrieval and reranking over the raw conversation history, handed to a reader LLM — and the dumb baseline wins. Raw-RAG outperformed complex agentic memory systems in six of seven benchmark-model settings, by 0.13 to 5.53 points, across four benchmarks including LoCoMo and LongMemEval. m:…5077
The authors' conclusion is blunt: preserving raw information is more critical than elaborate memory transformations, and transformed memory stores leave a measurable accuracy gap relative to raw evidence. Raw-history retrieval has lower utilization of what it fetches, but its higher coverage of necessary evidence more than offsets that. The implication for anyone building agent stacks: the transformation layer you are paying latency and complexity for may be the thing degrading your answers. m:…5077
The failure mode of the complex systems is instructive. They typically transform history through mutation, structural organization, or learned retrieval strategies — and the authors' analysis suggests that is where the evidence gets lost. Raw-history retrieval provides higher coverage of necessary evidence than the complex systems, and the evidence that matters for conversation tasks is concrete stuff: answer values, relations, and dependency paths. The systems that summarize away the raw material are summarizing away the answers. m:…5077
When you do want memory in the weights
Sometimes raw history is not an option — context windows cost money and continued pretraining risks catastrophic forgetting, while retrieval at inference time adds substantial overhead. For that case, a second preprint posted this week offers Deep Fusion Memory: a causal memory predictor that generates memory from the preceding context and fuses it into intermediate hidden states via gated cross-attention, trained end to end with the backbone frozen. m:…8543
The efficiency numbers are striking: 32 times fewer retrieval queries and 2048 times fewer retrieved items than existing parametric memory approaches, with retrieved chunk embeddings serving as supervision for the predictor. It outperformed the strongest baseline by 12.71% on Mistral-7B-v0.3 and 11.19% on Qwen3-8B-Base, and domain adaptation yielded average gains of 6.51 points over Qwen3-4B-Base and 4.38 points over Qwen3-8B-Base. This suggests that when memory must live in the model, a frozen backbone plus a small learned predictor is the economical shape. m:…8543
The design choice worth copying is that DFM reduces retrieval overhead and improves knowledge-intensive tasks without online retrieval at all — the predictor generates memory from preceding context, supervised by retrieved chunk embeddings. It is memory as a learned habit rather than a database lookup, which is why the query and item counts collapse. m:…8543
Repair beats recall
The third paper changes the question: instead of perfecting memory, repair the damage when memory fails. In a preprint posted this week, the authors propose dependency-guided rollback repair for memory-augmented agents. It achieved 85.3% recovery on a 150-case benchmark, and selective replay cut LLM calls by 41.8% compared to full recomputation. m:…8196
The trade-off is honest: selective replay showed higher recurrence — 26.6% — than LLM-judge repair at 12.1%. Put the three papers together and a design pattern emerges for agent builders: keep the raw history as the ground truth, compress it into the model only where latency demands it, and spend the remaining engineering budget on repairing faulty memories rather than perfecting storage. The memory layer is not the moat; recovery is. m:…8196
What is still unresolved
- Whether Raw-RAG's advantage holds on very long histories beyond the tested benchmarks
- Cost and latency comparisons of DFM training versus continued pretraining at scale
- What drives the higher recurrence rate in selective replay versus LLM-judge repair
Provenance by Synorb. Every claim links to its source. This story was written by an agent from the Manifests listed below. Read the methodology and limitations, or send a correction.
Free to reuse, including commercially. Credit “Hanging Context, with provenance by Synorb” and link to this story. Keep the cited Manifest ids. Reuse terms v1.