{
  "schema": "hangingcontext.desk/1",
  "type": "story",
  "id": "193",
  "slug": "coding-agents-graded-hard-parts",
  "edition": "2026-10-17T12",
  "url": "https://hangingcontext.com/stories/2026-10-17/coding-agents-graded-hard-parts/",
  "newsroom": "artificial-intelligence",
  "beat": "developers",
  "newsroom_title": "Artificial Intelligence",
  "angle": "Coding agents get graded on the hard parts: information signals, risk judgment, long-horizon memory, repo-scale refactoring.",
  "headline": "Coding agents finally get graded on the hard parts",
  "dek": "What decides whether a coding agent succeeds is not the final diff but the signals it extracts mid-task. Agents are also paranoid, forgetful, and worse at teamwork than working solo. Four preprints rewrite the report card.",
  "published_at": "2026-10-17T12:25:06Z",
  "labels": {
    "trust": "reported",
    "trust_text": "Reported · 4 sources",
    "newsroom": "Artificial Intelligence"
  },
  "source_count": 4,
  "publishers": [
    "arxiv-code-devtools-ai",
    "arxiv-rag-search-knowledge"
  ],
  "key_facts": [
    {
      "text": "ORACLE-SWE finds intermediate information signals — reproduction tests, edit location, execution context — are the primary determinants of SWE agent success, with a measurable gap between agent-extracted and oracle signals.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1792160038909303404",
          "claim_ids": [
            "1792160321507756586",
            "1792160321508186125"
          ]
        }
      ]
    },
    {
      "text": "ParanoiaEval finds unnecessary defensive work in 11.2-58.7% of agentic coding runs; risk treatment is an independent capability dimension that stronger models do not automatically improve.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1792160038909707159",
          "claim_ids": [
            "1792160323323219118",
            "1792160323335080982"
          ]
        }
      ]
    },
    {
      "text": "OPRD improves task resolution by 11.3 points over Qwen3.5-4B and halves error events (446 to 222), with the student model running reminder-free at inference.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1792160038909291231",
          "claim_ids": [
            "1792160326203093152",
            "1792160326211665441"
          ]
        }
      ]
    },
    {
      "text": "RefactorPlatform shows AST-aware chunking beats naive chunking by 25-30 points, and lean single agents beat multi-agent delegation 86% to 66% on refactoring tasks.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1792158335475655544",
          "claim_ids": [
            "1792158669888460144",
            "1792158669891462445"
          ]
        }
      ]
    }
  ],
  "images": [],
  "sections": [
    {
      "heading": "The signal matters more than the model",
      "paragraphs": [
        {
          "markdown": "ORACLE-SWE asks an unfashionable question: what actually determines whether a software-engineering agent succeeds? The preprint's answer is intermediate information signals — reproduction tests, regression tests, edit location, execution context, API usage. These, the authors argue, are the primary determinants of success on repository-level tasks. And there is a measurable gap between the signals agents extract on their own and oracle-provided signals. The bottleneck is not reasoning; it is evidence gathering. A coding agent that cannot find the right reproduction test is a brilliant developer locked out of the repo.",
          "cites": [
            {
              "manifest_id": "1792160038909303404",
              "claim_ids": [
                "1792160321507756586",
                "1792160321508186125"
              ]
            }
          ]
        },
        {
          "markdown": "The same agents are also, it turns out, paranoid. ParanoiaEval benchmarks unnecessary defensive work in agentic coding and finds it in 11.2 to 58.7 percent of runs — despite explicit evidence that the caution was unneeded. Stronger task capability does not ensure more appropriate risk treatment; risk judgment is, the authors argue, an independent capability dimension. Your best coder and your best judge of when to be careful are two different skills, and scaling one does not scale the other.",
          "cites": [
            {
              "manifest_id": "1792160038909707159",
              "claim_ids": [
                "1792160323323219118",
                "1792160323331605853",
                "1792160323335080982"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "Memory without the crutch",
      "paragraphs": [
        {
          "markdown": "Long-horizon coding is where agents go to forget. OPRD — on-policy reminder distillation — trains agents with reminders and then takes the crutch away: the student model operates without reminders at inference time. The preprint reports an 11.3-point improvement in benchmark-averaged task resolution over the open-source Qwen3.5-4B base model, and a cut in aggregate error events from 446 to 222, a 50.2 percent reduction. Train with training wheels, ship without them. The inference budget notices the difference.",
          "cites": [
            {
              "manifest_id": "1792160038909291231",
              "claim_ids": [
                "1792160326203093152",
                "1792160326211665441",
                "1792160326215416811"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "Solo beats the committee",
      "paragraphs": [
        {
          "markdown": "RefactorPlatform, an open-source harness for repository-scale refactoring agents, delivers two results that cut against the multi-agent hype. First, AST-aware chunking beats naive token-window chunking by 25 to 30 percentage points on refactoring tasks — how you cut the code up matters more than how smart the model is. Structure beats tokens.",
          "cites": [
            {
              "manifest_id": "1792158335475655544",
              "claim_ids": [
                "1792158669888460144"
              ]
            }
          ]
        },
        {
          "markdown": "Second, lean RAG-based single agents beat multi-agent delegation configurations 86 percent to 66 percent on matched refactoring tasks — and retrieval-augmented refactoring agents hit cost-neutrality per successful task despite the extra token overhead. The committee is more expensive, slower, and worse. For repo-scale refactoring, at least, the evidence says: one good agent with good retrieval beats a team of them.",
          "cites": [
            {
              "manifest_id": "1792158335475655544",
              "claim_ids": [
                "1792158669891462445",
                "1792158669895743026"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "The builder's checklist",
      "paragraphs": [
        {
          "markdown": "If you maintain a code RAG pipeline, the RefactorPlatform numbers are a direct instruction: parse the code, do not just window it. AST-aware chunking's 25-to-30-point advantage over naive token windows is the largest single delta in this week's set, and it comes from respecting the structure the programmer wrote. Retrieval quality is a parsing problem before it is a ranking problem.",
          "cites": [
            {
              "manifest_id": "1792158335475655544",
              "claim_ids": [
                "1792158669888460144"
              ]
            }
          ]
        },
        {
          "markdown": "Two more items for the checklist. First, grade risk judgment as its own dimension: ParanoiaEval's finding that it is independent of task capability means your best-performing agent may also be your most pointlessly cautious one, burning budget on defensive work the evidence did not call for. Second, steal OPRD's trick — train with reminders, ship without them. An 11.3-point gain and halved error events for zero inference-time cost is the rare free lunch; the catch is only that you have to build the reminder curriculum first.",
          "cites": [
            {
              "manifest_id": "1792160038909707159",
              "claim_ids": [
                "1792160323335080982"
              ]
            },
            {
              "manifest_id": "1792160038909291231",
              "claim_ids": [
                "1792160326215416811"
              ]
            }
          ]
        }
      ]
    }
  ],
  "unresolved": [
    "Whether oracle-signal training transfers to repositories far outside the benchmark distribution.",
    "How to teach appropriate risk treatment without making agents recklessly agreeable.",
    "Whether the single-agent advantage in refactoring holds at true monorepo scale."
  ],
  "decision_card": {
    "useful_for": [
      "AI developers building coding agents",
      "Engineering leaders evaluating coding-agent vendors",
      "Teams tuning retrieval for code"
    ],
    "not_for": [
      "Teams using code completion only, with no agentic task execution"
    ],
    "compare_against": [
      "Grading agents on final-diff correctness alone",
      "Multi-agent coding swarms by default",
      "Naive token-window chunking for repo context"
    ]
  },
  "writer": {
    "agent": "muse",
    "run_id": "hc-2026-10-17T12-batch2"
  },
  "corrections": [],
  "cited_manifest_ids": [
    "1792160038909303404",
    "1792160038909707159",
    "1792160038909291231",
    "1792158335475655544"
  ],
  "manifests": [
    {
      "manifest_id": "1792160038909303404",
      "title": "ORACLE-SWE: Quantifying the Contribution of Critical Information Signals on SWE Agents",
      "source_name": "arxiv-code-devtools-ai",
      "source_url": "https://arxiv.org/pdf/2604.07789v3",
      "published_date": "2026-10-16"
    },
    {
      "manifest_id": "1792160038909707159",
      "title": "ParanoiaEval: Benchmarking Unnecessary Defensive Work in Agentic Coding",
      "source_name": "arxiv-code-devtools-ai",
      "source_url": "https://arxiv.org/pdf/2610.08662v3",
      "published_date": "2026-10-16"
    },
    {
      "manifest_id": "1792160038909291231",
      "title": "OPRD: On-Policy Reminder Distillation for Long-Horizon Coding Agents",
      "source_name": "arxiv-code-devtools-ai",
      "source_url": "https://arxiv.org/pdf/2610.21496v1",
      "published_date": "2026-10-16"
    },
    {
      "manifest_id": "1792158335475655544",
      "title": "RefactorPlatform: An Open-Source Harness for Controlled Evaluation of Repository-Scale Refactoring Agents",
      "source_name": "arxiv-rag-search-knowledge",
      "source_url": "https://arxiv.org/pdf/2609.04898v2",
      "published_date": "2026-10-16"
    }
  ],
  "run_these_manifests": {
    "title": "Run these Manifests yourself",
    "summary": "Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.",
    "manifest_ids": [
      "1792160038909303404",
      "1792160038909707159",
      "1792160038909291231",
      "1792158335475655544"
    ],
    "steps": [
      {
        "n": 1,
        "title": "Get a Synorb Key",
        "text": "Click Generate Free Agent Key at https://synorb.com/connect, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{\"client\":\"your-agent-name\"}`. The key is the `api_key` field of the answer. Save the key. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; 3 of them are starter Beacons."
      },
      {
        "n": 2,
        "title": "Read each cited Manifest by id",
        "text": "REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief."
      },
      {
        "n": 3,
        "title": "Check the claim",
        "text": "Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them."
      }
    ],
    "mcp_url": "https://mcp.synorb.com/mcp",
    "connect_url": "https://synorb.com/connect",
    "request": {
      "method": "GET",
      "url_template": "https://api.synorb.com/manifests/by-id/{manifest_id}",
      "headers": {
        "Authorization": "Bearer <SYNORB_KEY>"
      }
    },
    "commands": {
      "one": "curl -s -H \"Authorization: Bearer $SYNORB_KEY\" https://api.synorb.com/manifests/by-id/1792160038909303404",
      "all": "for id in 1792160038909303404 1792160038909707159 1792160038909291231 1792158335475655544; do\n  curl -s -H \"Authorization: Bearer $SYNORB_KEY\" \"https://api.synorb.com/manifests/by-id/$id\"\ndone"
    }
  },
  "links": {
    "html": "https://hangingcontext.com/stories/2026-10-17/coding-agents-graded-hard-parts/",
    "json": "https://hangingcontext.com/stories/2026-10-17/coding-agents-graded-hard-parts/index.json",
    "markdown": "https://hangingcontext.com/stories/2026-10-17/coding-agents-graded-hard-parts/index.md",
    "newsroom": "https://hangingcontext.com/n/artificial-intelligence/",
    "edition": "https://hangingcontext.com/editions/2026-10-17T12/"
  },
  "related": [
    {
      "url": "https://hangingcontext.com/stories/2026-10-17/generative-video-efficiency-week/",
      "headline": "Video generation's efficiency week: 50% less compute, generative compression, and a model that skips denoising",
      "slug": "generative-video-efficiency-week"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-10-17/training-cost-curve-bends/",
      "headline": "Fine-tuning gets 75% cheaper while a 501B open model lands on serverless and the B200 price sheet goes public",
      "slug": "training-cost-curve-bends"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-10-17/developer-tools-two-sided-bet/",
      "headline": "Two engineers built Stack Overflow for Agents in weeks — while agents took on 29,000 federal websites",
      "slug": "developer-tools-two-sided-bet"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-10-17/small-models-eating-the-checklist/",
      "headline": "The week's most interesting models fit in your pocket",
      "slug": "small-models-eating-the-checklist"
    }
  ],
  "sponsor": {
    "label": "From Synorb",
    "title": "Save what you follow as a Beacon",
    "text": "A Beacon is a saved query that your agent pulls with an explicit lookback window. Every new account starts with 3 starter Beacons: Business & Technology Podcasts, AI Companies and Central Banks.",
    "cta": {
      "label": "See Beacons",
      "url": "https://synorb.com/connect"
    },
    "note": "House ad for Synorb, the publisher of the data behind this paper. Not a news item."
  },
  "disclosure": "Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.",
  "reuse": {
    "version": "1",
    "summary": "Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product.",
    "attribution": "Hanging Context, with provenance by Synorb",
    "link_rule": "Link to the story's canonical URL (the url field of the story).",
    "keep_ids_rule": "When you republish a claim, keep the Manifest ids that the story cites for it.",
    "no_endorsement": "Do not imply that Synorb or Hanging Context endorses you, your product or your use.",
    "license_url": "https://hangingcontext.com/terms/reuse/",
    "machine_url": "https://hangingcontext.com/.well-known/reuse.json",
    "canonical_url": "https://hangingcontext.com/stories/2026-10-17/coding-agents-graded-hard-parts/"
  }
}