Models that sleep, models that stall: continual learning gets both a new architecture and a hard exam

A model that sleeps

Google Research introduced a sleep framework for large language models that alternates between an active phase for processing inputs and a sleep phase for memory consolidation. The framework enables LLMs to continuously integrate new facts and enhance reasoning while mitigating catastrophic forgetting, using parameter expansion and a continuum memory hierarchy to prevent that forgetting. m:…4976

During the sleep phase, the model unlocks pre-allocated Mixture-of-Experts memory blocks and updates its weights using self-generated practice examples. The dreaming stage implements an autonomous self-improvement loop using random expert selection and curated rehearsal. The design idea is plain: a model should not only answer while awake, but should also consolidate what it learned while offline, the way a person reviews material after studying it. m:…4976

Google Research reports strong results on its chosen measures. The sleep framework demonstrated near-perfect scaling up to 10M tokens on the BABILong benchmark, outperforming GPT-4 and Titans, and achieved 48.9 percent on single-task and 46.2 percent on continued pre-training evaluations on SQuAD. The sleep-enhanced base model achieved accuracy scores of 79.2 percent, 69.0 percent and 46.1 percent on AIME 24/25 and HMMT 25 benchmarks. These are company-reported research results, and developers should treat them as a promising architecture to test, not as proof that forgetting is solved. m:…4976

The exam says agents barely learn

In a preprint, the Continual Learning Exam, or CLE, offers a sobering counterweight. CLE is a large-scale benchmark designed to measure experience-driven improvement in professional workflows, comprising 120 human-authored tasks organized into 600 levels, spanning 89 professional roles across 12 occupational domains, and developed by over 50 domain experts. The dataset includes approximately 24,000 files and 19 stateful applications. m:…7700

Diagnostic feedback and revision provide a median completion gain of 5.3 percentage points for LLM agents, which shows that feedback helps, but the ceiling is low: the best-performing system clears 47.7 percent of levels and completes all five levels in only 23.3 percent of tasks. An agent that improves with experience but still fails most full ladders is not yet a colleague that learns on the job; it is a system that gets a little better and then stalls. m:…7700

CLE can also be used to post-train LLMs to strengthen their continual learning capabilities, according to the authors, which turns the benchmark into a training resource rather than only a scoreboard. That dual use matters, because a benchmark that only measures failure helps less than one that also provides the material to improve. m:…7700

Two more mechanisms

In another preprint, C3L addresses explanation drift, where catastrophic forgetting manifests as both accuracy degradation and explanation drift: models maintain predictive accuracy while shifting attention from diagnostic features to spurious correlations. Pixel-level saliency regularization creates a self-referential drift loop by using the model own frozen saliency maps as supervision targets, so the model ends up supervising itself with its own drifted explanations. m:…5895

C3L anchors regularization to offline-discovered semantic concepts to break that loop, providing target independence from model drift and cross-instance semantic consistency, and employs fully automated concept discovery and spatial grounding, requiring no human-annotated concept sets. The concept-based objective combined with approximate LRP relevance conservation induces a crowding-out effect that suppresses spurious background correlations by forcing competition for relevance between concept and background regions. The authors report state-of-the-art performance across five benchmarks and over 20 baselines. m:…5895

In a further preprint, Nestor introduces nested sparsity in the attention operator to improve conditional computation, with sparsity that hierarchically composes with the sparsity induced by the Mixture-of-Experts operator. Nestor performs better than a conventional MoE transformer at equal compute and equal sparsity in language model pretraining, and the benefits persist as model size, dataset size and context length grow. Nestor is well-suited for continual pretraining, including adaptation to longer contexts and new domains, and code and checkpoints are available on GitHub. m:…8162

Validation is the gate

A final preprint, KEEL, addresses the practical side of agents that learn from experience: automatically collected skills and memories without validation propagate errors and corrupt agent knowledge. Skills and lessons are complementary components for reliable continual agent learning, the authors argue, and KEEL improves task success by 4.2 to 21.7 percentage points over the no-knowledge baseline on the SkillFlow benchmark. m:…2874

The lesson across all five papers is consistent. Architectures that consolidate memory, steer explanations toward real concepts, or sparsify attention may all help, but an agent that accumulates experience without validation will accumulate errors too. Developers building long-running agents should budget for the sleep phase, the validation step and the exam that shows whether learning actually happened, rather than assuming that more interaction automatically means more competence. m:…4976 m:…7700 m:…2874