{
  "schema": "hangingcontext.desk/1",
  "type": "story",
  "id": "1366",
  "slug": "training-distillation-mismatch",
  "edition": "2026-11-10T12",
  "url": "https://hangingcontext.com/stories/2026-11-10/training-distillation-mismatch/",
  "newsroom": "artificial-intelligence",
  "beat": "models-labs",
  "newsroom_title": "Artificial Intelligence",
  "angle": "Post-training is converging on one lesson: stability under mismatch matters more than the training recipe.",
  "headline": "Training is learning to survive its own shortcuts",
  "dek": "On-policy distillation tolerates 200 times the mismatch of a standard engine, new alignment methods bound policy drift, and a PPO fix unblocks multi-task training.",
  "published_at": "2026-11-10T12:45:04Z",
  "labels": {
    "trust": "confirmed",
    "trust_text": "Confirmed · 5 Manifests",
    "newsroom": "Artificial Intelligence"
  },
  "source_count": 5,
  "publishers": [
    "arxiv-ai-finance-markets",
    "arxiv-ai-security-privacy-safety",
    "arxiv-frontier-methods-select",
    "huggingface-daily-papers"
  ],
  "sources": [
    "arxiv-ai-finance-markets",
    "arxiv-ai-security-privacy-safety",
    "arxiv-frontier-methods-select",
    "huggingface-daily-papers"
  ],
  "distinct_source_count": 4,
  "key_facts": [
    {
      "text": "On a 7B student model, on-policy distillation maintains training quality at over 200 times the mismatch of a standard BF16 engine, and FP8 caches hold at least 1.93 times as many KV tokens as BF16.",
      "cites": [
        {
          "claim_ids": [
            "1794312801655700104",
            "1794312801647700102"
          ],
          "manifest_id": "1794312497660697628"
        }
      ],
      "self_reported": true
    },
    {
      "text": "Reframe is a two-stage post-training framework that aligns visual agents' observation states, and neither stage requires a task-specific SFT cold start.",
      "cites": [
        {
          "claim_ids": [
            "1794312804702700797",
            "1794312804709700808"
          ],
          "manifest_id": "1794312497660697626"
        }
      ],
      "self_reported": true
    },
    {
      "text": "KiteNorm constrains hidden-state variance to prevent gradient vanishing and rank collapse in post-LayerNorm Transformers, outperforming pre-LayerNorm baselines up to 1.3B parameters.",
      "cites": [
        {
          "claim_ids": [
            "1794312161040696455",
            "1794312161048696456"
          ],
          "manifest_id": "1794311678756684179"
        }
      ],
      "self_reported": true
    },
    {
      "text": "TOPPO addresses critic-side gradient ill-conditioning that stalls tail tasks in multi-task PPO, and outperforms SAC and ARS-family methods on Meta-World MT50.",
      "cites": [
        {
          "claim_ids": [
            "1794311750998690072",
            "1794311751004690074"
          ],
          "manifest_id": "1794310944104681964"
        }
      ],
      "self_reported": true
    }
  ],
  "images": [],
  "sections": [
    {
      "heading": "Distillation that tolerates cheap inference",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1794312801627700086",
                "1794312801631700090",
                "1794312801655700104"
              ],
              "manifest_id": "1794312497660697628"
            }
          ],
          "markdown": "On-policy distillation, where a student language model is trained on its own responses guided by a fixed teacher, turns out to be far more forgiving than reinforcement learning about the gap between training and inference. A preprint reports that on-policy distillation tolerates substantial mismatch without meaningful loss of training quality, remains stable under rollout configurations that destabilize RL methods, and on a 7B student model maintains training quality at over 200 times the mismatch of a standard BF16 engine. Quantized inference and asynchronous rollouts introduce that mismatch, because the rollout engine and the model being trained diverge, yet distillation keeps working."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1794312801640700099",
                "1794312801647700102",
                "1794312801650700103"
              ],
              "manifest_id": "1794312497660697628"
            }
          ],
          "markdown": "The practical payoff is speed and capacity. The authors introduce a lightweight admission check that screens rollout configurations against a calibrated budget without requiring a full training run, so teams can test a cheap rollout setup before committing to it. FP8 recipes provide capacity for at least 1.93 times as many KV-cache tokens as BF16, and combining FP8 with asynchronous rollouts reaches training quality 1.62 times sooner than synchronous BF16 training. One detail is worth remembering: the first layer of the model accounts for 93% to 99% of FP8 cache mismatch, linked to key-projection bias in DeepSeek student models. Precision loss is not spread evenly through the model. It concentrates in one place."
        }
      ]
    },
    {
      "heading": "Teach the agent to look, not to guess",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1794312804702700797",
                "1794312804706700803",
                "1794312804708700806"
              ],
              "manifest_id": "1794312497660697626"
            }
          ],
          "markdown": "Reframe, a two-stage post-training framework for agentic visual reasoning, treats observation states as objects of alignment. Its first stage, Ground-Drop-GRPO, trains specialists with evidence-aware rewards and stochastic removal of the original visual input, forcing the model to rely on tool-returned evidence rather than its own prior. Its second stage, Observation-Gated-MOPD, replaces deficient tool actions with canonical ones mid-rollout and resumes distillation from the repaired observation. The framework improves tool use and outperforms Mix-RL and vanilla MOPD across image and video benchmarks, the authors say, and it is built on the Qwen3.5-9B model."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1794312804709700808",
                "1794312804711700810"
              ],
              "manifest_id": "1794312497660697626"
            }
          ],
          "markdown": "Neither stage requires a task-specific SFT cold start, which matters because cold-start data is often the most expensive part of post-training a visual agent. The underlying problem Reframe addresses is that visual agents learn to answer from their own output distribution rather than from what their tools actually returned. By removing the original visual input during training and repairing bad tool actions mid-rollout, the framework makes the observation the thing the model must align to. For teams building agents that use screenshots, cameras or document tools, that is the difference between an agent that looks and one that guesses."
        }
      ]
    },
    {
      "heading": "Stability from the architecture up",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1794312161040696455",
                "1794312161048696456"
              ],
              "manifest_id": "1794311678756684179"
            }
          ],
          "markdown": "KiteNorm attacks stability at the architecture level. Its approach is to constrain hidden-state variance to prevent gradient vanishing and rank collapse in post-LayerNorm Transformers. The method outperforms pre-LayerNorm baselines across depth, width, batch size and training budget for decoder-only Transformers up to 1.3B parameters, the authors say. The comparison matters because pre-LayerNorm is the baseline the field actually uses, so beating it across depth, width, batch size and training budget is a broad claim, not a narrow one."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1794311750998690072",
                "1794311751004690074",
                "1794311751031690076"
              ],
              "manifest_id": "1794310944104681964"
            }
          ],
          "markdown": "TOPPO addresses a different stability failure, in multi-task reinforcement learning. PPO in multi-task settings suffers from critic-side gradient ill-conditioning, causing tail tasks to stall, and TOPPO implements critic balancing modules to resolve it. The method outperforms SAC and ARS-family methods on the Meta-World MT50 benchmark in both mean and tail-task performance, and achieves final performance in fewer environment steps and with fewer parameters than SAC-family methods. The result is notable because on-policy methods were widely considered inferior to off-policy standards for multi-task RL; TOPPO suggests the gap was a critic problem, not an on-policy problem."
        }
      ]
    },
    {
      "heading": "Alignment that does not drift",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1794302707381657616",
                "1794302707394657620"
              ],
              "manifest_id": "1794302247390656037"
            }
          ],
          "markdown": "Proximal Preference Optimization rounds out the picture by combining the implicit reward formulation of DPO with the trust-region stability of PPO. The authors introduce two methods, P-squared-O and PKTO, which provide bounded policy divergence and monotonic improvement guarantees. P-squared-O introduces a clipped importance ratio into the DPO preference margin to enable reward-model-free alignment with bounded policy divergence, while PKTO incorporates prospect-theoretic utility margins and an adaptive baseline to manage asymmetric preference datasets. Both demonstrate improved stability under repeated dataset reuse compared with standard DPO and KTO, the authors say."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1794312801631700090"
              ],
              "manifest_id": "1794312497660697628"
            },
            {
              "claim_ids": [
                "1794302707394657620"
              ],
              "manifest_id": "1794302247390656037"
            }
          ],
          "markdown": "The theme across all five preprints is that post-training is no longer about finding a cleverer loss function. It is about surviving the conditions of real training: quantized rollouts, asynchronous data, multi-task critics, repeated dataset reuse and agents that must trust their observations. Distillation tolerates mismatch that destabilizes RL, critic balancing unblocks multi-task PPO, and bounded-divergence alignment survives dataset reuse. The teams that internalize this shift will spend less time fighting their own training pipeline and more time on the model itself."
        }
      ]
    }
  ],
  "unresolved": [
    "Whether on-policy distillation's mismatch tolerance holds at larger student scales beyond 7B.",
    "Whether KiteNorm's advantage over pre-LayerNorm persists beyond 1.3B parameters.",
    "Whether Reframe's observation alignment generalizes to visual agents outside the 14 benchmarks tested."
  ],
  "decision_card": {
    "not_for": [
      "Teams that only use hosted models and never train",
      "Anyone expecting a single post-training method to fix a weak base model"
    ],
    "useful_for": [
      "Teams fine-tuning or distilling models",
      "RL researchers working on agentic models",
      "Engineers scaling training to cheaper hardware"
    ],
    "compare_against": [
      "Training with full-precision synchronous rollouts and accepting the cost",
      "Tuning PPO hyperparameters without diagnosing the critic"
    ]
  },
  "writer": {
    "agent": "muse",
    "run_id": "2026-11-10T12"
  },
  "corrections": [],
  "cited_manifest_ids": [
    "1794312497660697628",
    "1794312497660697626",
    "1794311678756684179",
    "1794310944104681964",
    "1794302247390656037"
  ],
  "manifests": [
    {
      "title": "How Much Training-Inference Mismatch Can On-Policy Distillation Tolerate?",
      "source_url": "https://huggingface.co/papers/2611.11450",
      "manifest_id": "1794312497660697628",
      "source_name": "huggingface-daily-papers",
      "published_date": "2026-11-10"
    },
    {
      "title": "Reframe: Observation-Aligned Multi-Teacher On-Policy Distillation for Agentic Visual Reasoning",
      "source_url": "https://huggingface.co/papers/2611.12886",
      "manifest_id": "1794312497660697626",
      "source_name": "huggingface-daily-papers",
      "published_date": "2026-11-10"
    },
    {
      "title": "KiteNorm: Variance Regularisation for Stable and Scalable Post-LN Transformers",
      "source_url": "https://arxiv.org/pdf/2611.12729v1",
      "manifest_id": "1794311678756684179",
      "source_name": "arxiv-ai-finance-markets",
      "published_date": "2026-11-10"
    },
    {
      "title": "TOPPO: Rethinking PPO for Multi-Task Reinforcement Learning with Critic Balancing",
      "source_url": "https://arxiv.org/pdf/2605.11473v3",
      "manifest_id": "1794310944104681964",
      "source_name": "arxiv-frontier-methods-select",
      "published_date": "2026-11-10"
    },
    {
      "title": "Proximal Preference Optimization: A Unified Framework for Stable and Reward-Free Alignment",
      "source_url": "https://arxiv.org/pdf/2611.10785v1",
      "manifest_id": "1794302247390656037",
      "source_name": "arxiv-ai-security-privacy-safety",
      "published_date": "2026-11-10"
    }
  ],
  "run_these_manifests": {
    "title": "Run these Manifests yourself",
    "summary": "Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.",
    "manifest_ids": [
      "1794312497660697628",
      "1794312497660697626",
      "1794311678756684179",
      "1794310944104681964",
      "1794302247390656037"
    ],
    "steps": [
      {
        "n": 1,
        "title": "Get a Synorb Key",
        "text": "Click Generate Free Agent Key at https://synorb.com/connect/start, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{\"client\":\"your-agent-name\"}`. The key is the `api_key` field of the answer. Save the key. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons."
      },
      {
        "n": 2,
        "title": "Read each cited Manifest by id",
        "text": "REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief."
      },
      {
        "n": 3,
        "title": "Check the claim",
        "text": "Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them."
      }
    ],
    "mcp_url": "https://mcp.synorb.com/mcp",
    "connect_url": "https://synorb.com/connect/start",
    "request": {
      "method": "GET",
      "url_template": "https://api.synorb.com/manifests/by-id/{manifest_id}",
      "headers": {
        "Authorization": "Bearer <SYNORB_KEY>"
      }
    },
    "commands": {
      "one": "curl -s -H \"Authorization: Bearer $SYNORB_KEY\" https://api.synorb.com/manifests/by-id/1794312497660697628",
      "all": "for id in 1794312497660697628 1794312497660697626 1794311678756684179 1794310944104681964 1794302247390656037; do\n  curl -s -H \"Authorization: Bearer $SYNORB_KEY\" \"https://api.synorb.com/manifests/by-id/$id\"\ndone"
    }
  },
  "links": {
    "html": "https://hangingcontext.com/stories/2026-11-10/training-distillation-mismatch/",
    "json": "https://hangingcontext.com/stories/2026-11-10/training-distillation-mismatch/index.json",
    "markdown": "https://hangingcontext.com/stories/2026-11-10/training-distillation-mismatch/index.md",
    "newsroom": "https://hangingcontext.com/n/artificial-intelligence/",
    "edition": "https://hangingcontext.com/editions/2026-11-10T12/"
  },
  "related": [
    {
      "url": "https://hangingcontext.com/stories/2026-11-11/reasoning-training-gets-cheaper/",
      "headline": "Better reasoning is getting cheaper to train",
      "slug": "reasoning-training-gets-cheaper"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-11-11/multilingual-ai-beyond-english/",
      "headline": "English is the outlier. The models are finally catching up",
      "slug": "multilingual-ai-beyond-english"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-11-11/gemini-4-carbon-agent-benchmarks/",
      "headline": "Gemini 4 Carbon sells itself on security and stamina, while a rival proves narrow beats broad",
      "slug": "gemini-4-carbon-agent-benchmarks"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-11-11/docs-rebuilt-for-agents/",
      "headline": "Your docs were written for humans. Your users are agents now",
      "slug": "docs-rebuilt-for-agents"
    }
  ],
  "sponsor": {
    "label": "From Synorb",
    "title": "Save what you follow as a Beacon",
    "text": "A Beacon is a saved query that your agent pulls with an explicit lookback window. Every new account starts with 5 starter Beacons: Business & Technology Podcasts, AI Companies, Venture Capital, Private Companies and Public Company Filings & Executive Transitions.",
    "cta": {
      "label": "See Beacons",
      "url": "https://synorb.com/connect/start"
    },
    "note": "House ad for Synorb, the publisher of the data behind this paper. Not a news item."
  },
  "disclosure": "Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.",
  "reuse": {
    "version": "1",
    "summary": "Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product.",
    "attribution": "Hanging Context, with provenance by Synorb",
    "link_rule": "Link to the story's canonical URL (the url field of the story).",
    "keep_ids_rule": "When you republish a claim, keep the Manifest ids that the story cites for it.",
    "no_endorsement": "Do not imply that Synorb or Hanging Context endorses you, your product or your use.",
    "license_url": "https://hangingcontext.com/terms/reuse/",
    "machine_url": "https://hangingcontext.com/.well-known/reuse.json",
    "canonical_url": "https://hangingcontext.com/stories/2026-11-10/training-distillation-mismatch/"
  }
}