Models are teaching themselves now, and picking up things nobody taught them

No new data required

On-Policy Self-Training, described in a preprint, reformulates self-distillation as a self-training framework driven by self-generated question-answer pairs and privileged reasoning demonstrations. Guided solely by coarse, high-level topic descriptions, the model synthesizes its own curriculum of queries and reasoning traces without access to any human-curated prompts or answers, unlike existing approaches that rely on ground-truth pairs or human-curated problem sets. m:…8716

The quality control is a pseudo-verification filter: it keeps a problem only if the model can reproduce its answer in a fresh context and under a paraphrase. The authors report consistent self-improvement on standard language benchmarks and visual question answering, and that models trained this way retain capabilities on domains they were not self-trained on, while outperforming alternative training strategies. m:…8716

A second preprint applies the same idea inside looped language models. LoopOPD provides dense supervision without external teachers or privileged information by using a frozen terminal loop policy as a compute-privileged teacher for intermediate loop students, and D-LoopOPD extends this by continually refreshing the terminal loop teacher as shared parameters update. The authors report improvements in mathematical reasoning, general reasoning, and code generation despite training only on mathematical data. m:…5763

The catch: preferences travel too

EleutherAI researchers investigated a phenomenon they call subliminal learning, in which a student model acquires a teacher's preference through training data that appears semantically unrelated to that preference. They found that cross-model transfer can survive differences in initialization, data order, pretraining corpus, model size, and model architecture between teacher and student. m:…3019

Transmitting a preference and acquiring one are distinct abilities that can develop separately during pretraining, the researchers report, and the signal can hide in statistics as thin as number frequencies: number frequencies alone can carry a substantial portion of the preference signal in hard-label distillation, and synthetic sequences retained 40 to 92% of real-sequence transfer across four animal preferences and three Pythia-160M variants. m:…3019

The transfer is broad, not a curiosity of one setup: on-policy soft-label distillation showed positive transfer in 26 of 30 cross-model pairs, extending to larger Pythia models. Removing animal-name tokens from the loss sharply reduced transfer but did not eliminate it in the same-model setting, and neither low-rank adapters nor adaptive optimizers are required for the effect. m:…3019

What to watch

Read together, the three pieces describe one mechanism with two faces. Self-training without human data works because a model can generate, check, and learn from its own output. Subliminal learning works for the same reason: a student can absorb a teacher's tendencies from data that carries no visible trace of them. The verification filter is therefore the interesting part of OPST, not the generation step, because it is the only thing standing between self-improvement and self-contamination. m:…8716 m:…3019

For developers, the practical implication is that evaluation has to check for transferred preferences, not just benchmark gains. A model that improves on the benchmarks while quietly inheriting a teacher's leanings has not necessarily gotten better in the sense its users care about, and the EleutherAI results suggest that checking only the obvious channels, such as shared initialization or matching data, will miss the transfer. m:…3019

LoopOPD's design is worth noting for builders: the teacher is the model's own terminal loop, not an external model, which removes the need to serve a separate teacher at training time. And the EleutherAI results show the effect arrives early: in Pythia, the improvement in transfer appears within 4,000 pretraining steps in the soft-label setting, which means preferences can start moving long before a model looks finished. m:…5763 m:…3019