Distillation inherits the teacher's failures
On-policy distillation propagates incomplete answers, reward hacks, and lost token signals. Four preprints map the failure modes and the filters that fix them.
- In a preprint, on-policy distillation underperforms supervised fine-tuning by 7.9 points on Geometry3K, with the gap explained by incomplete teacher responses; starting OPD from an SFT student cuts its completion rate from 76% to 62%, while filtering teacher demonstrations for completeness recovers much of SFT's advantage. Self-reported m:…7010
- In a preprint, jointly optimizing heterogeneous objectives in VLMs introduces cross-capability gradient conflicts and direct reward optimization entangles specialized knowledge with reward-hacking biases; the decoupled SCOPED framework outperforms backbone and mixed-capability RL across eight multimodal benchmarks. Self-reported m:…4785
- In a preprint, heterogeneous tokenizers in on-policy distillation lose teacher signal wherever vocabularies disagree; SimCT recovers the discarded supervision by comparing teacher and student over short multi-token continuations, beating shared-vocabulary OPD and cross-tokenizer baselines on math and code. Self-reported m:…2588
- In a preprint on search-agent distillation, the first search turn accounts for 51.1% of total absolute information gain but receives only 18.0% of the supervision budget, and 43.7% of queries have teacher rewards that disagree in sign with post-retrieval information utility; information-gain-calibrated distillation lets a 4B student outperform its 8B teacher on MMSearch and BrowseComp-VL. Self-reported m:…7530
The teacher's mistakes are the curriculum
In a preprint, the authors compare on-policy distillation with supervised fine-tuning in vision-language models and find that distillation is data-efficient but fragile: it works only when the teacher remains reliable on student-generated trajectories. On Geometry3K, on-policy distillation underperforms SFT by 7.9 points, and most of that gap is explained by incomplete responses the student inherits from the teacher. m:…7010
The damage compounds. Starting distillation from an SFT student cuts its completion rate from 76% to 62%, effectively erasing the completion behavior the fine-tuning had installed. The fix the authors identify is almost embarrassingly simple: filtering teacher demonstrations for completeness alone recovers much of SFT's advantage. Nobody is auditing the teacher's homework, and the student is learning the gaps. m:…7010
Gradients fight, rewards hack
A second preprint, introducing the SCOPED framework, shows that the problem runs deeper than bad demonstrations. Jointly optimizing heterogeneous objectives in unified vision-language models introduces cross-capability gradient conflicts, and direct reward optimization entangles specialized knowledge with reward-hacking biases. The student is not just copying the teacher's mistakes; the training dynamics are manufacturing new ones. m:…4785
The authors decouple capability training from unified model optimization: train specialized experts, then use reward-guided reflection to distill procedural knowledge into the student without keeping explicit skill modules at inference. The decoupled approach consistently outperforms the backbone and mixed-capability RL across eight multimodal benchmarks. Separation of concerns, it turns out, applies to distillation too. m:…4785
The plumbing loses signal
A third preprint finds signal loss in the machinery itself. When teacher and student use different tokenizers, exact token matching fails wherever vocabularies disagree, silently discarding teacher supervision at those positions. SimCT recovers it by expanding the supervision space to short, jointly tokenizable multi-token continuations, so the teacher-student distinction survives the vocabulary mismatch. m:…2588
The recovered supervision shows up where it is hardest to fake: SimCT outperforms shared-vocabulary on-policy distillation and representative cross-tokenizer baselines on math and code benchmarks. Anyone distilling across heterogeneous architectures, which is to say anyone distilling a frontier open model into a custom student, is currently training on a supervision stream with holes in it. m:…2588
Supervise what the evidence was worth
The fourth preprint names the subtlest failure: evidence-blind supervision. Standard on-policy distillation for search agents supervises student queries based on preceding context, without accounting for the utility of the retrieved evidence. The first search turn accounts for 51.1% of total absolute information gain but receives only 18.0% of the supervision budget, and 43.7% of search queries carry teacher rewards that disagree in sign with post-retrieval information utility. The teacher is grading the question while ignoring what the answer was worth. m:…7530
Calibrating supervision with post-retrieval information gain reallocates training toward information-critical turns, and the payoff is a genuine upset: a 4B student outperforming its 8B teacher on MMSearch and BrowseComp-VL. The meta-lesson across all four preprints, and this is reasoning, is that distillation is not compression. It is inheritance, and inheritances need auditing: filter the demonstrations, decouple the objectives, fix the token plumbing, and grade what the evidence was worth. m:…7530
For builders, the practical checklist is now concrete: completeness-filter every teacher demonstration, separate capability training from unified optimization, verify supervision survives tokenizer boundaries, and weight training by information gain rather than turn order. Distillation pipelines that skip these steps are not saving compute. They are manufacturing a cheaper, worse teacher. m:…7010 m:…7530
What is still unresolved
- How to filter teacher demonstrations for completeness and reliability at frontier scale without human review
- Whether decoupled expert-then-distill training becomes standard practice for multimodal agents
Provenance by Synorb. Every claim links to its source. This story was written by an agent from the Manifests listed below. Read the methodology and limitations, or send a correction.
Free to reuse, including commercially. Credit “Hanging Context, with provenance by Synorb” and link to this story. Keep the cited Manifest ids. Reuse terms v1.