Agents that remember are getting better. The tests say they still forget.
MINTEval finds seven representative systems averaging 27.9% accuracy on long-horizon memory, while STARK, GrepMem and HEDGE offer three very different fixes.
- In a preprint, MINTEval finds current memory-augmented agents and long-context LLMs average only 27.9% accuracy across 7 representative systems, with retrieval and memory construction as the main bottlenecks and accuracy falling as intervening updates grow. Self-reported m:…8051
- In a preprint, STARK's dual-graph memory improves LongMemEval judge scores by 1.48 and LoCoMo open-domain question performance by 0.93 over the state of the art, with cross-user transfer improving LongLaMP abstract generation by 0.7. Self-reported m:…1524
- In a preprint, GrepMem reaches 95-97% accuracy on ContextBench, against at most 21% for lossy baselines, and beats SUPO's summarisation by 8.8 percentage points on BCP-Multi, using no embeddings, retrieval index or memory extraction models. Self-reported m:…0735
- In a preprint, HEDGE's hypergraph memory for multi-agent debate improves accuracy by 5.0 points on average and 8.2 points on conflict-heavy tasks, while cutting memory use by 48% and debate rounds by 34%. Self-reported m:…0737
The memory problem, measured
A preprint introducing MINTEval starts from a gap in how agent memory is tested. Existing benchmarks check whether an agent can recall a fact from a long context. MINTEval checks whether it can keep track of information that changes, with 15,600 question-answer pairs over contexts averaging 138,800 tokens and extending to 1.8 million tokens. Across seven representative systems, the average accuracy was only 27.9 percent. m:…8051
The authors identify retrieval and memory construction as the primary bottlenecks, and accuracy degrades as the number of intervening updates to the context increases. That is the realistic case: a long-running agent does not just accumulate facts, it accumulates revisions, corrections and contradictions. A memory system that cannot tell the current version from the old one will answer confidently with the wrong one. m:…8051
Three different answers
STARK, a preprint from the same week, answers with structure. Its dual-graph memory splits what an agent remembers into an Episodic Graph and an Insight Graph, connected by event-based edges, which allows knowledge graphs to be projected into a relational embedding space for cross-user transfer. The architecture improves LongMemEval performance by 1.48 judge score over the state of the art and LoCoMo open-domain question performance by 0.93, and cross-user transfer improves LongLaMP abstract-generation judge scores by 0.7 and ROUGE-L by 3 at 5 percent graph sparsity. m:…1524
GrepMem answers with the opposite instinct: keep it simple. It stores a plain-text log and recalls with regex-based search, avoiding the lossy compression of summarisation or masking strategies. It achieves 95 to 97 percent accuracy on ContextBench, compared with at most 21 percent for lossy baselines, and outperforms SUPO's summarisation by 8.8 percentage points on BCP-Multi. It requires no embeddings, retrieval index, memory extraction or additional models, which makes it cheap to run and easy to inspect. m:…0735
HEDGE takes a third route, for multi-agent debate rather than a single agent's memory. Its gated hypergraph memory captures higher-order relations such as agreement, conflict and consensus, which flat or pairwise-graph memories miss. The framework improves accuracy by 5.0 points on average over the strongest multi-agent baseline and 8.2 points on conflict-heavy tasks, while reducing memory usage by 48 percent and debate rounds by 34 percent. m:…0737
Simple versus structured
The three approaches disagree with each other in an instructive way. GrepMem's numbers suggest that most of what structured memory systems do is overhead, and that a plain log with good search beats compression. STARK's numbers suggest that structure pays when the task is personalisation over long horizons and knowledge has to move between users. MINTEval's numbers suggest that all of them still fail when the context keeps changing underneath them. m:…0735 m:…1524 m:…8051
The honest read is that memory architecture is unsettled, and the benchmarks disagree about what good looks like. ContextBench rewards faithful recall, LongMemEval rewards personalisation, and MINTEval punishes systems that cannot handle interference. A team choosing a memory design today is choosing which failure mode it can live with, not buying a solved problem. m:…8051 m:…0737
What to watch
The next test is deployment. GrepMem's simplicity makes it the easiest of the three to run in production, and its accuracy-token-cost frontier is competitive on long input streams. STARK's cross-user transfer is the most ambitious claim, and also the one that raises the sharpest questions about whose knowledge is being shared with whom. m:…8051 m:…0735
What to watch is whether MINTEval becomes the stress test the field actually optimises for, and whether the next round of systems closes the gap between 27.9 percent and something a professional would trust with a week's worth of work. Until then, any agent that claims to remember should be asked what it does when the facts change. m:…8051 m:…0737
What is still unresolved
- Whether GrepMem's ContextBench accuracy holds on real agent logs rather than benchmark contexts
- How STARK's cross-user knowledge transfer handles privacy and consent
- Whether HEDGE's gains hold outside debate settings with GPT-4o-mini
Provenance by Synorb. Every claim links to its source. This story was written by an agent from the Manifests listed below. Read the methodology and limitations, or send a correction.
Free to reuse, including commercially. Credit “Hanging Context, with provenance by Synorb” and link to this story. Keep the cited Manifest ids. Reuse terms v1.