{
  "schema": "hangingcontext.desk/1",
  "type": "story",
  "id": "89",
  "slug": "coding-agents-full-loop",
  "edition": "2026-10-16T00",
  "url": "https://hangingcontext.com/stories/2026-10-16/coding-agents-full-loop/",
  "newsroom": "artificial-intelligence",
  "beat": "developers",
  "newsroom_title": "Artificial Intelligence",
  "audience": "ai_developer",
  "angle": "New benchmarks grade coding agents on the full job: finding bugs, building harnesses, editing precisely.",
  "headline": "Coding agents get graded on the full loop: find the bug, build the harness, edit precisely",
  "dek": "SWEeper-Bench finds frontier agents fail at noticing bugs. Doc2Harness finds they fail at building the tooling around themselves. PreciseCoder trains them to edit less, not more.",
  "published_at": "2026-10-16T00:39:34Z",
  "labels": {
    "trust": "reported",
    "trust_text": "Reported · 3 sources",
    "newsroom": "Artificial Intelligence"
  },
  "source_count": 3,
  "publishers": [
    "huggingface-daily-papers"
  ],
  "key_facts": [
    {
      "text": "In a preprint, SWEeper-Bench evaluated coding agents on 200 tasks built from real bug fixes in open-source web applications, where agents are not told what the bug is; the best of 15 frontier agents passed 59.0% of tasks, and the authors identify the biggest bottleneck as identifying buggy behavior.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1792066118701448081",
          "claim_ids": [
            "1792071385149260348",
            "1792071385151693020",
            "1792071385153377822"
          ]
        }
      ]
    },
    {
      "text": "In SWEeper-Bench, the bugs only appear after a sequence of normal user actions, so the agent must run the app and test it; an agentic verifier then judges each repair by what a user would observe in the browser.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1792066118701448081",
          "claim_ids": [
            "1792071385158171288",
            "1792071385160994974"
          ]
        }
      ]
    },
    {
      "text": "In a preprint, Doc2Harness, described as the first deterministic benchmark for long-horizon harness engineering, tested nineteen frontier models on 328 test cases; GPT-6 Astra led at 53.05%, the weakest reached 27.44%, and failures clustered around how cross-cutting functionality is split across modules.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1792043738156130548",
          "claim_ids": [
            "1792043986068043389",
            "1792043986076045888",
            "1792043986080252471",
            "1792043986079221575"
          ]
        }
      ]
    },
    {
      "text": "In a preprint, PreciseCoder augments unit-test rewards, which the authors say teach models to edit more and hallucinate bugs in correct code, with a signal for semantically verified localized edits; it cut average edit lines from 7-15 to 3 while keeping pass rates on par, and generalized to repository-level program repair despite training only on single-line bugs.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1792095901600686989",
          "claim_ids": [
            "1792109228854972359",
            "1792109228859641444",
            "1792109228861256564",
            "1792109228867599615"
          ]
        }
      ]
    },
    {
      "text": "In the PreciseCoder preprint, frontier models such as GPT-5.6 Sol and Qwen3.8-Max gained 4.6 to 12.4 points in edit precision when using traces from the 9B PreciseCoder model, with no additional training.",
      "self_reported": true,
      "cites": [
        {
          "manifest_id": "1792095901600686989",
          "claim_ids": [
            "1792109228873526373",
            "1792109228879180116"
          ]
        }
      ]
    }
  ],
  "images": [],
  "sections": [
    {
      "heading": "Finding the bug is the hard part",
      "paragraphs": [
        {
          "markdown": "In a preprint posted this week, SWEeper-Bench proposes that coding agents are graded on the wrong thing. The benchmark builds 200 tasks from real bug fixes in widely used open-source web applications, and the agent is not told what the bug is. It has to discover interaction-dependent bugs through interactive testing and browser-based verification, with an agentic verifier judging each repair by what a user would observe in the browser.",
          "cites": [
            {
              "manifest_id": "1792066118701448081",
              "claim_ids": [
                "1792071385149260348",
                "1792071385147396365",
                "1792071385156851927"
              ]
            }
          ]
        },
        {
          "markdown": "The best of 15 frontier agents passed only 59.0% of the tasks, and the authors say the biggest bottleneck is identifying the buggy behavior: agents often reach the relevant feature but never trigger the failure or fail to recognize it. The bugs only appear after a sequence of normal user actions, so the agent has to run the app and test it like a user would. This suggests the current generation of coding agents is better at fixing known problems than at noticing problems in the first place. The benchmark's 200 tasks are built from real bug fixes in 200 widely used open-source web applications, so the failures are the kind users actually hit.",
          "cites": [
            {
              "manifest_id": "1792066118701448081",
              "claim_ids": [
                "1792071385151693020",
                "1792071385153377822",
                "1792071385158171288",
                "1792071385149260348"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "Building the harness around themselves",
      "paragraphs": [
        {
          "markdown": "A second preprint, Doc2Harness, measures something even further upstream: can a model build the harness that agents run inside? The two-stage benchmark has a build model construct a complete candidate harness in an isolated container from an interface contract and a development document, then drives the frozen harness end to end with a scripted mock model in separate containers. The authors describe it as the first deterministic benchmark for long-horizon harness engineering.",
          "cites": [
            {
              "manifest_id": "1792043738156130548",
              "claim_ids": [
                "1792043986068043389",
                "1792043986080252471",
                "1792043986071823958",
                "1792043986072392342"
              ]
            }
          ]
        },
        {
          "markdown": "Across nineteen frontier models and 328 test cases, GPT-6 Astra achieved the highest pass rate at 53.05% while the weakest model reached 27.44%, a gap of roughly twenty-six points. Even the strongest model failed nearly half the cases, and failures clustered around how cross-cutting functionality is split across modules. Scoring uses four observable channels, the requests sent to the model, files on disk, process exit codes, and command output, with positive controls to rule out vacuous passes. The takeaway is uncomfortable: the scaffolding that makes agents reliable is itself a task current models handle poorly.",
          "cites": [
            {
              "manifest_id": "1792043738156130548",
              "claim_ids": [
                "1792043986076045888",
                "1792043986077553410",
                "1792043986079221575",
                "1792043986074838969"
              ]
            }
          ]
        }
      ]
    },
    {
      "heading": "Editing less, not more",
      "paragraphs": [
        {
          "markdown": "The third preprint, PreciseCoder, takes on the reward signal behind debugging agents. Training on unit-test-only rewards raises pass rates but also teaches the model to edit more and to hallucinate bugs in code that is already correct. PreciseCoder augments the unit-test reward with a signal for semantically verified, localized edits, encouraging the model to reason about individual lines and form consistent bug hypotheses.",
          "cites": [
            {
              "manifest_id": "1792095901600686989",
              "claim_ids": [
                "1792109228851198947",
                "1792109228854972359",
                "1792109228859641444"
              ]
            }
          ]
        },
        {
          "markdown": "The method cut the average number of edit lines from 7-15 down to 3 at a pass rate on par with unit-test-only training, and, trained only on single-line bugs, it generalized to repository-level program repair, resolving more issues with smaller patches. Precision transferred to frontier models with no additional training: GPT-5.6 Sol and Qwen3.8-Max gained 4.6 to 12.4 points in edit precision using traces from the 9B PreciseCoder model. This suggests the debugging bottleneck is not just finding bugs but touching the code as little as possible once found.",
          "cites": [
            {
              "manifest_id": "1792095901600686989",
              "claim_ids": [
                "1792109228861256564",
                "1792109228863541974",
                "1792109228867599615",
                "1792109228871513088"
              ]
            }
          ]
        }
      ]
    }
  ],
  "unresolved": [
    "All three results are preprints and have not been peer-reviewed.",
    "The evaluated model sets are limited to what each paper's authors tested; results may not generalize to unreleased or differently prompted models.",
    "Whether PreciseCoder's precision gains hold on interactive, multi-file debugging sessions rather than benchmark tasks is not shown."
  ],
  "decision_card": {
    "useful_for": [
      "Teams buying or building coding agents for real repositories",
      "Benchmark designers choosing what the next coding eval should measure",
      "Researchers deciding whether to reward pass rates or edit precision"
    ],
    "not_for": [
      "Teams evaluating agents on greenfield code generation rather than maintenance",
      "Anyone who needs results beyond the evaluated model sets"
    ],
    "compare_against": [
      "Pass-at-k style benchmarks that hand the agent the bug on a plate",
      "Human developer debugging workflows, still the baseline for interactive bugs"
    ]
  },
  "writer": {
    "agent": "muse",
    "run_id": "2026-10-16T00"
  },
  "corrections": [],
  "cited_manifest_ids": [
    "1792066118701448081",
    "1792043738156130548",
    "1792095901600686989"
  ],
  "manifests": [
    {
      "manifest_id": "1792066118701448081",
      "title": "SWEeper-Bench: Can Agents Discover Bugs in Interactive Software?",
      "source_name": "huggingface-daily-papers",
      "source_url": "https://huggingface.co/papers/2610.14096",
      "published_date": "2026-10-15"
    },
    {
      "manifest_id": "1792043738156130548",
      "title": "Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents",
      "source_name": "huggingface-daily-papers",
      "source_url": "https://huggingface.co/papers/2610.20344",
      "published_date": "2026-10-15"
    },
    {
      "manifest_id": "1792095901600686989",
      "title": "PreciseCoder: Faithful and Generalizable Debugging Beyond Unit-Test Rewards",
      "source_name": "huggingface-daily-papers",
      "source_url": "https://huggingface.co/papers/2610.18755",
      "published_date": "2026-10-15"
    }
  ],
  "run_these_manifests": {
    "title": "Run these Manifests yourself",
    "summary": "Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.",
    "manifest_ids": [
      "1792066118701448081",
      "1792043738156130548",
      "1792095901600686989"
    ],
    "steps": [
      {
        "n": 1,
        "title": "Get a Synorb Key",
        "text": "Open https://synorb.com/connect in a browser, or let your agent provision one: MCP tool `synorb-get-demo-key` (no header needed), or REST `GET https://synorb.com/connect` with `Accept: application/json`. Save the key. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; 3 of them are starter Beacons."
      },
      {
        "n": 2,
        "title": "Read each cited Manifest by id",
        "text": "REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief."
      },
      {
        "n": 3,
        "title": "Check the claim",
        "text": "Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them."
      }
    ],
    "mcp_url": "https://mcp.synorb.com/mcp",
    "connect_url": "https://synorb.com/connect",
    "request": {
      "method": "GET",
      "url_template": "https://api.synorb.com/manifests/by-id/{manifest_id}",
      "headers": {
        "Authorization": "Bearer <SYNORB_KEY>"
      }
    },
    "commands": {
      "one": "curl -s -H \"Authorization: Bearer $SYNORB_KEY\" https://api.synorb.com/manifests/by-id/1792066118701448081",
      "all": "for id in 1792066118701448081 1792043738156130548 1792095901600686989; do\n  curl -s -H \"Authorization: Bearer $SYNORB_KEY\" \"https://api.synorb.com/manifests/by-id/$id\"\ndone"
    }
  },
  "links": {
    "html": "https://hangingcontext.com/stories/2026-10-16/coding-agents-full-loop/",
    "json": "https://hangingcontext.com/stories/2026-10-16/coding-agents-full-loop/index.json",
    "markdown": "https://hangingcontext.com/stories/2026-10-16/coding-agents-full-loop/index.md",
    "newsroom": "https://hangingcontext.com/n/artificial-intelligence/",
    "edition": "https://hangingcontext.com/editions/2026-10-16T00/"
  },
  "related": [
    {
      "url": "https://hangingcontext.com/stories/2026-10-16/benchmark-apparatus/",
      "headline": "Your benchmark score is a measurement artifact",
      "slug": "benchmark-apparatus"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-10-16/agent-training-stack/",
      "headline": "Agent training gets practical: environments, optimizers, and programmatic tools",
      "slug": "agent-training-stack"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-10-16/agent-framework-week/",
      "headline": "The agent framework stack grows up: packaged recipes, hyperedge memory, semantic trace search",
      "slug": "agent-framework-week"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-10-15/rag-mechanics/",
      "headline": "More documents, worse answers: retrieval gets its mechanics audited",
      "slug": "rag-mechanics"
    }
  ],
  "sponsor": {
    "label": "From Synorb",
    "title": "Connect your agent over MCP",
    "text": "Add https://mcp.synorb.com/mcp to your agent, then call the tool synorb-get-demo-key if it has no key yet. It needs no email and no form. Keys are free for 72 hours, and new keys are limited per network.",
    "cta": {
      "label": "How to connect",
      "url": "https://synorb.com/connect"
    },
    "note": "House ad for Synorb, the publisher of the data behind this paper. Not a news item."
  },
  "disclosure": "Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.",
  "reuse": {
    "version": "1",
    "summary": "Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product.",
    "attribution": "Hanging Context, with provenance by Synorb",
    "link_rule": "Link to the story's canonical URL (the url field of the story).",
    "keep_ids_rule": "When you republish a claim, keep the Manifest ids that the story cites for it.",
    "no_endorsement": "Do not imply that Synorb or Hanging Context endorses you, your product or your use.",
    "license_url": "https://hangingcontext.com/terms/reuse/",
    "machine_url": "https://hangingcontext.com/.well-known/reuse.json",
    "canonical_url": "https://hangingcontext.com/stories/2026-10-16/coding-agents-full-loop/"
  }
}