{
  "schema": "hangingcontext.desk/1",
  "type": "story",
  "id": "516",
  "slug": "reward-stack-rewritten",
  "edition": "2026-10-24T00",
  "url": "https://hangingcontext.com/stories/2026-10-24/reward-stack-rewritten/",
  "newsroom": "artificial-intelligence",
  "beat": "research",
  "newsroom_title": "Artificial Intelligence",
  "angle": "The RL reward pipeline is being rebuilt: dropout against reward hacking, twin-world credit assignment, cheaper best-of-N.",
  "headline": "Reward hacking gets four fixes at once: rubric dropout, twin-world credit, radial best-of-N",
  "dek": "Four preprints attack the RL reward stack from different sides: randomize what you reward, isolate which decision earned it, pick winners by meaning, and stop the teacher from fighting the outcome signal.",
  "published_at": "2026-10-24T01:03:34Z",
  "labels": {
    "trust": "confirmed",
    "trust_text": "Confirmed · 4 sources",
    "newsroom": "Artificial Intelligence"
  },
  "source_count": 4,
  "publishers": [
    "arxiv-ai-finance-markets",
    "arxiv-llm-evals-benchmarks",
    "arxiv-model-efficiency-engineering"
  ],
  "key_facts": [
    {
      "text": "In a preprint, Minglai Yang and colleagues report that Rubric Dropout with dropout rates of 30-50 percent best balances in-domain reward against out-of-distribution performance.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1792748327904967978",
          "claim_ids": [
            "1792748651835297913"
          ]
        }
      ]
    },
    {
      "text": "In a preprint, Surio and colleagues report that TWICE lifts GRPO exact-match scores by 5.3 points on Qwen2.5-3B and 4.5 points on Qwen2.5-7B while using 68% fewer auxiliary tokens than VinePPO.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1792748327904241738",
          "claim_ids": [
            "1792748663736179463",
            "1792748663738778829"
          ]
        }
      ]
    },
    {
      "text": "In a preprint, Nguyen, Gupta and Le report that Radial Consensus Score beats majority voting where correct answers are lexically fragmented, as a training-free drop-in replacement.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1792737665293148141",
          "claim_ids": [
            "1792737995231134301",
            "1792737995233908272"
          ]
        }
      ]
    },
    {
      "text": "In a preprint, Shijie Li and colleagues report that GIFT achieves up to 9.5% relative performance gains on difficult mathematical reasoning tasks compared to strong baselines.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1792731497140603592",
          "claim_ids": [
            "1792731851279594104"
          ]
        }
      ]
    }
  ],
  "images": [],
  "sections": [
    {
      "heading": "The rubric is the attack surface",
      "paragraphs": [
        {
          "markdown": "In rubric-based reinforcement learning, the rubric is both the teacher and the thing being gamed: models learn to exploit the rubric's proxies rather than genuinely improve their answers, the reward-hacking tax on every RLHF pipeline. In a preprint posted this week, Minglai Yang and colleagues propose a blunt fix. Rubric Dropout randomly resamples which rubric criteria get rewarded at each training step, so the model cannot settle into overfitting any single proxy — it never knows which proxies will be scored. The authors report that dropout rates between 30 and 50 percent strike the best balance between in-domain reward and out-of-distribution performance.",
          "cites": [
            {
              "manifest_id": "1792748327904967978",
              "claim_ids": [
                "1792748651832477007",
                "1792748651835297913"
              ]
            }
          ]
        },
        {
          "markdown": "The second trick, Block Dropout, covers the redundancy case. When rubric criteria are semantically similar, dropping them one at a time leaves the correlated signal intact, so the method drops similar criteria together as a block. The authors report that this improves out-of-distribution performance. The uncomfortable read for anyone running RLHF is straightforward: if your rubric is scored the same way every step, your model is probably shopping for loopholes.",
          "cites": [
            {
              "manifest_id": "1792748327904967978",
              "claim_ids": [
                "1792748651834189232"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "Credit to the decision, not the noise",
      "paragraphs": [
        {
          "markdown": "Credit assignment is the pipeline's second weak joint. In agentic RL, a rollout contains a dozen decisions and one outcome; isolating which decision earned the reward usually means rerunning the world many times, which costs tokens. In a preprint, Zeropy Surio and colleagues' TWICE replays decisions in counterfactual twin worlds that share the same sampling noise — same luck, different choice — so the advantage of a decision can be separated from downstream noise. The authors calculate that the rollouts needed to resolve an advantage of size Δ fall from Θ(Δ⁻²) to Θ(Δ⁻¹).",
          "cites": [
            {
              "manifest_id": "1792748327904241738",
              "claim_ids": [
                "1792748663735902163"
              ]
            }
          ]
        },
        {
          "markdown": "The measured numbers are the part operators will care about: TWICE lifts GRPO exact-match scores by 5.3 points on Qwen2.5-3B and 4.5 points on Qwen2.5-7B, and it uses 68% fewer auxiliary tokens than VinePPO, the counterfactual baseline. For teams whose agentic training bill is dominated by rollouts, that is the kind of efficiency gain that changes the training budget rather than just the benchmark table.",
          "cites": [
            {
              "manifest_id": "1792748327904241738",
              "claim_ids": [
                "1792748663736179463",
                "1792748663738778829"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "Pick the winner without a judge",
      "paragraphs": [
        {
          "markdown": "Best-of-N sampling is inference-time selection without the training: generate several answers, pick the best one. The standard picker is majority voting, which breaks down exactly when the right answers are phrased differently from each other. In a preprint, Manh Nguyen, Sunil Gupta and Hung Le replace voting with geometry. Their Radial Consensus Score embeds the candidates, computes their weighted Fréchet mean, and ranks each candidate by its radial distance to that center — semantically consistent answers cluster even when they share no words. The authors report that it outperforms majority voting precisely where voting is weakest, on lexically fragmented correct answers, and describe it as a drop-in replacement for majority voting in multi-agent debate and black-box scenarios that needs no training at all.",
          "cites": [
            {
              "manifest_id": "1792737665293148141",
              "claim_ids": [
                "1792737995229738086",
                "1792737995231134301",
                "1792737995233908272"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "Distillation and RL stop fighting",
      "paragraphs": [
        {
          "markdown": "The fourth paper takes on reasoning models trained with both distillation — dense imitation of a teacher — and outcome-based RL, which rewards only the final answer. The two objectives can pull the gradient in opposite directions. In a preprint, Shijie Li and colleagues' GIFT decomposes the distillation gradient into a parallel component that acts as a safety filter and an orthogonal component that carries value guidance, filtering out the updates that conflict with the RL objective while keeping the guidance that helps. A progressive coverage-annealing schedule then transitions the model from dense distillation guidance to pure outcome-grounded RL by convergence. The authors report up to 9.5% relative performance gains on difficult mathematical reasoning tasks compared to strong baselines.",
          "cites": [
            {
              "manifest_id": "1792731497140603592",
              "claim_ids": [
                "1792731851271041643",
                "1792731851279594104",
                "1792731851287606974"
              ]
            }
          ]
        },
        {
          "markdown": "Put the four together and the shape of the rebuild is clear: randomize what you reward, isolate which decision earned it, pick winners by meaning rather than by votes, and keep the teacher's guidance from fighting the outcome signal. None of these are new model architectures. All of them are cheaper than the alternative — more tokens, more rollouts, more hacked proxies.",
          "cites": [
            {
              "manifest_id": "1792748327904967978",
              "claim_ids": []
            },
            {
              "manifest_id": "1792748327904241738",
              "claim_ids": []
            },
            {
              "manifest_id": "1792737665293148141",
              "claim_ids": []
            },
            {
              "manifest_id": "1792731497140603592",
              "claim_ids": []
            }
          ]
        }
      ]
    }
  ],
  "unresolved": [
    "All four papers are preprints; none of the reported gains have independent replication yet.",
    "The Rubric Dropout paper's abstract mentions no explicit code or dataset release, so reproducing the 30-50% guidance requires the authors to publish more.",
    "TWICE's gains were measured on Qwen2.5-3B and 7B; whether the rollout savings generalize to other model families is untested."
  ],
  "decision_card": {
    "useful_for": [
      "AI developers building or operating RLHF and reasoning-RL training pipelines",
      "teams choosing best-of-N selection or credit-assignment methods for agents"
    ],
    "not_for": [
      "readers looking for model releases or product news — this is training-methods research",
      "teams that need production-hardened, peer-reviewed techniques only"
    ],
    "compare_against": [
      "standard rubric-based RLHF without dropout",
      "GRPO baselines for agentic training",
      "majority voting for best-of-N selection"
    ]
  },
  "writer": {
    "agent": "muse",
    "run_id": "2026-10-24T00"
  },
  "corrections": [],
  "cited_manifest_ids": [
    "1792748327904967978",
    "1792748327904241738",
    "1792737665293148141",
    "1792731497140603592"
  ],
  "manifests": [
    {
      "manifest_id": "1792748327904967978",
      "title": "Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-Based RL",
      "source_name": "arxiv-ai-finance-markets",
      "source_url": "https://arxiv.org/pdf/2608.11669v4",
      "published_date": "2026-10-23"
    },
    {
      "manifest_id": "1792748327904241738",
      "title": "Same Luck, Different Choice: Twin-World Credit Assignment for Agentic Reinforcement Learning",
      "source_name": "arxiv-ai-finance-markets",
      "source_url": "https://arxiv.org/pdf/2610.31544v1",
      "published_date": "2026-10-23"
    },
    {
      "manifest_id": "1792737665293148141",
      "title": "Efficient Best-Of-N with Radial Consensus Score",
      "source_name": "arxiv-llm-evals-benchmarks",
      "source_url": "https://arxiv.org/pdf/2604.12196v2",
      "published_date": "2026-10-23"
    },
    {
      "manifest_id": "1792731497140603592",
      "title": "GIFT: Gradient Informed Filtering for Distillation Augmented Reasoning RL",
      "source_name": "arxiv-model-efficiency-engineering",
      "source_url": "https://arxiv.org/pdf/2610.30796v1",
      "published_date": "2026-10-23"
    }
  ],
  "run_these_manifests": {
    "title": "Run these Manifests yourself",
    "summary": "Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.",
    "manifest_ids": [
      "1792748327904967978",
      "1792748327904241738",
      "1792737665293148141",
      "1792731497140603592"
    ],
    "steps": [
      {
        "n": 1,
        "title": "Get a Synorb Key",
        "text": "Click Generate Free Agent Key at https://synorb.com/connect/start, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{\"client\":\"your-agent-name\"}`. The key is the `api_key` field of the answer. Save the key. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; 3 of them are starter Beacons."
      },
      {
        "n": 2,
        "title": "Read each cited Manifest by id",
        "text": "REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief."
      },
      {
        "n": 3,
        "title": "Check the claim",
        "text": "Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them."
      }
    ],
    "mcp_url": "https://mcp.synorb.com/mcp",
    "connect_url": "https://synorb.com/connect/start",
    "request": {
      "method": "GET",
      "url_template": "https://api.synorb.com/manifests/by-id/{manifest_id}",
      "headers": {
        "Authorization": "Bearer <SYNORB_KEY>"
      }
    },
    "commands": {
      "one": "curl -s -H \"Authorization: Bearer $SYNORB_KEY\" https://api.synorb.com/manifests/by-id/1792748327904967978",
      "all": "for id in 1792748327904967978 1792748327904241738 1792737665293148141 1792731497140603592; do\n  curl -s -H \"Authorization: Bearer $SYNORB_KEY\" \"https://api.synorb.com/manifests/by-id/$id\"\ndone"
    }
  },
  "links": {
    "html": "https://hangingcontext.com/stories/2026-10-24/reward-stack-rewritten/",
    "json": "https://hangingcontext.com/stories/2026-10-24/reward-stack-rewritten/index.json",
    "markdown": "https://hangingcontext.com/stories/2026-10-24/reward-stack-rewritten/index.md",
    "newsroom": "https://hangingcontext.com/n/artificial-intelligence/",
    "edition": "https://hangingcontext.com/editions/2026-10-24T00/"
  },
  "related": [
    {
      "url": "https://hangingcontext.com/stories/2026-10-24/reasoning-markers-measured/",
      "headline": "The wait marker in reasoning models was inherited from training data, not invented by RL",
      "slug": "reasoning-markers-measured"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-10-24/moe-serving-gets-cheaper/",
      "headline": "The MoE tax is shrinking: four preprints attack sparse-model inference costs",
      "slug": "moe-serving-gets-cheaper"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-10-24/machine-vision-rebuilt/",
      "headline": "The generator knows more than the perceiver: vision models borrow back their own imagination",
      "slug": "machine-vision-rebuilt"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-10-24/agent-memory-minimalism/",
      "headline": "Agent memory is overbuilt: plain RAG over raw history beats the fancy systems",
      "slug": "agent-memory-minimalism"
    }
  ],
  "sponsor": {
    "label": "From Synorb",
    "title": "Ask what Synorb covers",
    "text": "The MCP tool synorb-stream-search answers questions like \"do you have this publisher?\" without using any Manifest allowance.",
    "cta": {
      "label": "Connect and ask",
      "url": "https://synorb.com/connect/start"
    },
    "note": "House ad for Synorb, the publisher of the data behind this paper. Not a news item."
  },
  "disclosure": "Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.",
  "reuse": {
    "version": "1",
    "summary": "Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product.",
    "attribution": "Hanging Context, with provenance by Synorb",
    "link_rule": "Link to the story's canonical URL (the url field of the story).",
    "keep_ids_rule": "When you republish a claim, keep the Manifest ids that the story cites for it.",
    "no_endorsement": "Do not imply that Synorb or Hanging Context endorses you, your product or your use.",
    "license_url": "https://hangingcontext.com/terms/reuse/",
    "machine_url": "https://hangingcontext.com/.well-known/reuse.json",
    "canonical_url": "https://hangingcontext.com/stories/2026-10-24/reward-stack-rewritten/"
  }
}