Agent Memory Is Distorting Decisions, Not Just Storing Them

Group decisions go wrong

In a preprint, researchers test how language-model groups make decisions when each agent holds only part of the evidence, a setup known as a hidden-profile configuration. They report that memory selection by rarity and relevance both fail in group decisions, because decoy inclusion and social-proof bias distort what the group ends up considering. Selecting by relevance lost 23 to 31 points to rarest-first selection in hidden-profile configurations, yet rarest-first selection was no better than random selection on average. The benchmark behind the study, CrossProfile, consists of 60 templates and 480 tasks, and the study evaluated Qwen2.5-7B, Llama-3.1-8B, and two DeepSeek models. m:…0650

Two further results from the same preprint are harder to dismiss as a quirk of one filter. Full transcripts reduced accuracy in group decisions compared with providing no notes at all, which means more context was actively worse than none in that setup. Language models also exhibited social-proof bias by giving extra weight to facts attributed to many agents, and removing or randomising attribution tags raised accuracy by 13 to 24 points in classic configurations. The practical implication is that a memory system does not merely retrieve facts; it also signals which facts look popular or well-attested, and that signal can mislead the group. m:…0650

The memory tax

In a second preprint, researchers introduce a framework called Memory Tax to assess the net value of agent memory systems, and their headline finding is blunt: dedicated memory systems often underperform naive retrieval-augmented generation, even at higher costs. Memory tax arises because memory processes change how raw information is organised and accessed, and memory processes may remove or obscure evidence in the course of distilling it. The authors conducted a large-scale evaluation across multiple benchmarks and model backbones, and they say they will open-source the code and results. The argument is not that memory is useless, but that its benefit must be measured net of what the distillation step destroys. m:…1511

That framing reframes a common engineering choice. Adding a dedicated memory layer is often presented as a pure upgrade over retrieval, on the grounds that summarised history is cleaner and cheaper to use. The preprint suggests the opposite can happen: the summary step can delete the very detail that would have changed a decision, while still charging the cost of building and maintaining the memory layer. A team that does not measure against a naive retrieval baseline therefore cannot tell whether its memory system is helping, and the preprint argues that baseline is the comparison that matters. m:…1511

Habits are harder still

In a third preprint, researchers introduce HABIT-Bench to test whether agent memory systems can infer implicit user habits from sparse, long-term dialogue evidence. The benchmark uses synthetic user histories spanning hundreds of sessions, covers four domains including food, finance, software, and travel, and includes temporary exceptions, changes in habits, and suggestions that users have not accepted. Eight memory methods were evaluated using Qwen3-235B-A22B and GLM-5.3-Flash. The best-performing memory method achieved only 33.61 percent macro accuracy across the four domains, against 25 percent for random guessing on four-choice tasks. m:…6173

The preprint adds a result that rules out the simplest excuse. Performance remained limited even when annotated evidence was supplied directly, and finding relevant history was only part of the challenge for language-model agents. In other words, the bottleneck is not only retrieval. The harder part is distinguishing a recurring habit from a one-off exception or a changed preference when the evidence is weak and spread over hundreds of sessions. That is a judgment problem, not a storage problem, and it is the same kind of judgment problem that the group-decision preprint exposes when attribution and filtering distort which facts a group trusts. m:…6173

What builders should test

Taken together, the three preprints argue for a different default in agent design. Before adding a memory layer, test it against naive retrieval on the same tasks and count the cost, because the memory-tax preprint shows the dedicated layer can lose that comparison. Before trusting a group decision, test whether attribution tags or relevance filters are steering the outcome, because the group-decision preprint shows both can move accuracy by double-digit points. And before claiming an agent knows a user, test it on habits with exceptions and changes, because HABIT-Bench shows that even supplied evidence does not make that inference easy. m:…1511 m:…0650 m:…6173

None of this means agents should have no memory. It means memory should be treated as a hypothesis to be validated, with failure modes named in advance: evidence lost in distillation, popularity mistaken for truth, and a habit inferred from a single exception. The preprints give builders concrete tests for each failure mode, and the numbers they report set a sober baseline for what current methods actually achieve when history is long, evidence is thin, and decisions are shared. m:…1511 m:…0650 m:…6173