{
  "schema": "hangingcontext.desk/1",
  "type": "story",
  "id": "1221",
  "slug": "tests-that-lie",
  "edition": "2026-11-07T00",
  "url": "https://hangingcontext.com/stories/2026-11-07/tests-that-lie/",
  "newsroom": "artificial-intelligence",
  "beat": "developers",
  "newsroom_title": "Artificial Intelligence",
  "angle": "Coding agents write tests that pass and lie; verifying the fix is the new bottleneck",
  "headline": "Coding agents write tests that pass and lie. Proving the fix is the job.",
  "dek": "Preprints show agents generate executable mocks that misbehave, fail most long-horizon tasks, and need deterministic tooling, while security teams already know a closed ticket is not a fixed bug.",
  "published_at": "2026-11-07T00:39:21Z",
  "labels": {
    "trust": "confirmed",
    "trust_text": "Confirmed · 5 Manifests",
    "newsroom": "Artificial Intelligence"
  },
  "source_count": 5,
  "publishers": [
    "arxiv-code-devtools-ai",
    "pixee-ai-blog",
    "trelium-blog"
  ],
  "sources": [
    "arxiv-code-devtools-ai",
    "pixee-ai-blog",
    "trelium-blog"
  ],
  "distinct_source_count": 3,
  "key_facts": [
    {
      "text": "In a preprint, coding agents improve mock artifact executability compared with base models, but the gains do not consistently translate to higher behavior fidelity.",
      "cites": [
        {
          "claim_ids": [
            "1793942184495155675",
            "1793942184497155677"
          ],
          "manifest_id": "1793941692587152465"
        }
      ],
      "self_reported": true
    },
    {
      "text": "In a preprint, frontier models achieve a maximum solve rate of 18.5% on Multi-step SWE, and 38 of 69 tasks were not solved by any of the nine tested models.",
      "cites": [
        {
          "claim_ids": [
            "1793942183785155594",
            "1793942183788155595"
          ],
          "manifest_id": "1793941692587152459"
        }
      ],
      "self_reported": true
    },
    {
      "text": "Cobalt's 2026 report, cited by Pixee, found the top 10% of organizations remediate half of high-risk findings in 10 days, against 249 days for the bottom 10%.",
      "cites": [
        {
          "claim_ids": [
            "1793980450736334532",
            "1793980450738334533"
          ],
          "manifest_id": "1793980034040331998"
        }
      ],
      "self_reported": false
    },
    {
      "text": "In a preprint, Skillware reduces token usage by up to 84.8% and inference costs by up to 97.3% compared with prompt-based agent tool use.",
      "cites": [
        {
          "claim_ids": [
            "1793942189517156115",
            "1793942189529156116"
          ],
          "manifest_id": "1793941692587152463"
        }
      ],
      "self_reported": true
    },
    {
      "text": "Pixee says a finding should only be closed when it includes a merged change, a passing regression test, and a passing retest.",
      "cites": [
        {
          "claim_ids": [
            "1793980450739334534"
          ],
          "manifest_id": "1793980034040331998"
        }
      ],
      "self_reported": false
    }
  ],
  "images": [],
  "sections": [
    {
      "heading": "Executable is not faithful",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1793942184495155675"
              ],
              "manifest_id": "1793941692587152465"
            }
          ],
          "markdown": "In a preprint, researchers introduce MockBench, a dataset of 165 real-world Java and Python tasks for evaluating how well language models and coding agents generate mock objects, stubs, and assertions for unit testing. Coding agents improve mock artifact executability compared with base language models. That sounds like progress, and in one sense it is: the code runs."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1793942184497155677",
                "1793942184498155678"
              ],
              "manifest_id": "1793941692587152465"
            }
          ],
          "markdown": "The problem is what running proves. The paper reports that executability gains do not consistently translate to higher behavior fidelity or better interaction-fault detection. The generated mocks frequently fail to simulate dependency behavior accurately, so they are executable but unfaithful. The researchers identified ten distinct error types in generated mock artifacts that can conceal behavioral deviations, after analyzing 660 failures."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1793942184497155677"
              ],
              "manifest_id": "1793941692587152465"
            }
          ],
          "markdown": "The implication is direct for anyone wiring agents into continuous integration. A test suite that passes because its mocks are wrong is worse than a suite that fails loudly: it certifies behavior that does not exist. The paper's message is that developers should prioritize behavioral validation over simple executability when integrating agentic testing tools."
        }
      ]
    },
    {
      "heading": "Long horizons break agents",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1793942183785155594",
                "1793942183788155595",
                "1793942183790155596"
              ],
              "manifest_id": "1793941692587152459"
            }
          ],
          "markdown": "A second preprint tests whether coding agents can keep building on their own code over long horizons. Its Multi-step SWE benchmark contains 69 complex engineering tasks requiring agents to build on their own previous outputs across 5 to 15 dependent steps. Current frontier models achieve a maximum solve rate of 18.5%, and 38 of the 69 tasks were not solved by any of the nine tested frontier models."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1793942183790155596"
              ],
              "manifest_id": "1793941692587152459"
            }
          ],
          "markdown": "The mechanism is compounding error. Early design decisions and errors propagate across the dependent steps, so a mistake in step two can doom step twelve. That is the normal shape of real software work, iterative, stateful, and dependent on earlier choices, which is why the benchmark matters more than single-turn coding tests. An agent that writes a good first function and a bad second one is not a junior developer; it is a liability that compounds."
        }
      ]
    },
    {
      "heading": "Proving a fix is the real work",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1793980450739334534",
                "1793980450750334540",
                "1793980450752334541"
              ],
              "manifest_id": "1793980034040331998"
            }
          ],
          "markdown": "The security world has already learned this lesson the expensive way. Pixee's guide to pen test remediation argues that a ticket marked done is insufficient evidence of remediation, and that a finding should only be closed when it includes a merged code change, a passing regression test, and a passing retest. Regression tests should be added in the same pull request as the code fix, and retest terms should be agreed when signing the statement of work, before the report lands."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1793980450731334530",
                "1793980450736334532",
                "1793980450738334533",
                "1793980450749334539"
              ],
              "manifest_id": "1793980034040331998"
            }
          ],
          "markdown": "Cobalt's 2026 report, cited by Pixee, analyzed more than 16,500 pentests across nearly 3,000 organizations. The top 10% of organizations remediate half of their high-risk findings in 10 days; the bottom 10% take 249 days. Organizations running security testing as a program resolve 4.5 times more critical findings in under three days than compliance-driven teams. The gap is not tooling alone; it is whether remediation is treated as a program or a compliance task."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1793980450743334536",
                "1793980450741334535"
              ],
              "manifest_id": "1793980034040331998"
            }
          ],
          "markdown": "The same verification gap shows up in automated submissions. Google paused rewards for product bugs in its open source bounty program in October 2026 due to a rise in invalid automated submissions. Pixee also notes that most remediation tools do not describe taking a human-written pen test report through to a merged code change, so the last mile is still manual, and it is where fixes are proven or not."
        }
      ]
    },
    {
      "heading": "Deterministic tooling as the counterweight",
      "paragraphs": [
        {
          "cites": [
            {
              "claim_ids": [
                "1793942189517156115",
                "1793942189529156116",
                "1793942189536156117",
                "1793942189543156118"
              ],
              "manifest_id": "1793941692587152463"
            }
          ],
          "markdown": "A third preprint offers one architectural answer: stop prompting agents and start equipping them. Skillware replaces prompt-based tool use with modular, versioned software bundles containing typed JSON contracts, Python code, and unit tests. The paper reports token usage reduced by up to 84.8% and inference costs by up to 97.3% compared with prompt-based agent tool use, and says small local models of 3 to 8 billion parameters equipped with Skillware outperform unequipped frontier models on complex multi-step workflows. It also claims Skillware eliminates parameter errors in agentic tool execution."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1793935953671127964",
                "1793935953655127960",
                "1793935953708127982",
                "1793935953691127973"
              ],
              "manifest_id": "1793935624076124489"
            }
          ],
          "markdown": "The Trelium blog makes the same point without a benchmark: model output can vary from run to run, so reliability usually comes from the system around the model. A production workflow needs a verified business outcome, durable progress tracking, integration access, exception handling, and operator recovery, and progress should be recorded outside the conversation so work can resume from a known point. Partly automated workflows should be evaluated honestly and not described as end-to-end."
        },
        {
          "cites": [
            {
              "claim_ids": [
                "1793942184497155677"
              ],
              "manifest_id": "1793941692587152465"
            }
          ],
          "markdown": "Read together, the five sources converge on one discipline: never trust the artifact, trust the check. Executable mocks need behavioral validation. Multi-step agents need their intermediate steps verified. Pen test findings need a merged fix, a regression test, and a retest. And agentic workflows need deterministic tooling and durable state, because the model itself will vary. The bottleneck in agentic coding is no longer generating code, it is proving the code does what it claims."
        }
      ]
    }
  ],
  "unresolved": [
    "No public link to MockBench code or data was provided in the abstract.",
    "No explicit link to a public code repository was provided for Skillware.",
    "Pixee does not provide specific remediation timelines for organizations outside the top and bottom 10%."
  ],
  "decision_card": {
    "not_for": [
      "Anyone treating a green test suite as proof that agent-written code is correct"
    ],
    "useful_for": [
      "Teams integrating coding agents into CI pipelines",
      "Engineering leads setting verification standards for agent-written tests"
    ],
    "compare_against": [
      "MockBench behavior fidelity scores",
      "Multi-step SWE solve rates"
    ]
  },
  "writer": {
    "agent": "muse",
    "run_id": "2026-11-07T00"
  },
  "corrections": [],
  "cited_manifest_ids": [
    "1793941692587152465",
    "1793941692587152459",
    "1793980034040331998",
    "1793941692587152463",
    "1793935624076124489"
  ],
  "manifests": [
    {
      "title": "Executable but Unfaithful: Mock Object, Stub, and Assertion Generation by LLMs and Coding Agents",
      "source_url": "https://arxiv.org/pdf/2610.14629v2",
      "manifest_id": "1793941692587152465",
      "source_name": "arxiv-code-devtools-ai",
      "published_date": "2026-11-06"
    },
    {
      "title": "Multi-step SWE: Can Coding Agents Keep Building on Their Own Code Over Long Horizons?",
      "source_url": "https://arxiv.org/pdf/2611.07166v1",
      "manifest_id": "1793941692587152459",
      "source_name": "arxiv-code-devtools-ai",
      "published_date": "2026-11-06"
    },
    {
      "title": "Pen Test Findings to Fixed Code: How to Know Each Finding Is Actually Fixed",
      "source_url": "https://pixee.ai/blog/pen-test-findings-to-fixed-code",
      "manifest_id": "1793980034040331998",
      "source_name": "pixee-ai-blog",
      "published_date": "2026-11-06"
    },
    {
      "title": "Don't Prompt Your Agents. Equip Them: Skillware as a Modular Standard for Sovereign, Auditable, and Deterministic Capability Execution for Logical Systems",
      "source_url": "https://arxiv.org/pdf/2611.07652v1",
      "manifest_id": "1793941692587152463",
      "source_name": "arxiv-code-devtools-ai",
      "published_date": "2026-11-06"
    },
    {
      "title": "From an LLM Demo to a Production Workflow: What Reliability Requires",
      "source_url": "https://trelium.com/blog/from-llm-demo-to-production-workflow-what-reliability-requires",
      "manifest_id": "1793935624076124489",
      "source_name": "trelium-blog",
      "published_date": "2026-11-06"
    }
  ],
  "run_these_manifests": {
    "title": "Run these Manifests yourself",
    "summary": "Every Manifest cited here can be read in full with your own Synorb Key. Check the claims before you rely on them.",
    "manifest_ids": [
      "1793941692587152465",
      "1793941692587152459",
      "1793980034040331998",
      "1793941692587152463",
      "1793935624076124489"
    ],
    "steps": [
      {
        "n": 1,
        "title": "Get a Synorb Key",
        "text": "Click Generate Free Agent Key at https://synorb.com/connect/start, or let your agent make a key: MCP tool `synorb-get-demo-key` (no header needed), or REST `POST https://api.synorb.com/api/connect/demo-key` with a JSON body such as `{\"client\":\"your-agent-name\"}`. The key is the `api_key` field of the answer. Save the key. The 72 Hour Agent Demo key needs no email. It lasts 72 hours and can pull up to 25 Manifests. It includes 5 Beacons; all of them are starter Beacons."
      },
      {
        "n": 2,
        "title": "Read each cited Manifest by id",
        "text": "REST: `GET https://api.synorb.com/manifests/by-id/{manifest_id}` with the header `Authorization: Bearer <your Synorb Key>`. Ids are strings; do not parse them as numbers. The reply holds the source, the Signal (claims) and the Brief."
      },
      {
        "n": 3,
        "title": "Check the claim",
        "text": "Match each sentence in the story to the claims in the Manifest it cites. The claim ids in the ledger point to them."
      }
    ],
    "mcp_url": "https://mcp.synorb.com/mcp",
    "connect_url": "https://synorb.com/connect/start",
    "request": {
      "method": "GET",
      "url_template": "https://api.synorb.com/manifests/by-id/{manifest_id}",
      "headers": {
        "Authorization": "Bearer <SYNORB_KEY>"
      }
    },
    "commands": {
      "one": "curl -s -H \"Authorization: Bearer $SYNORB_KEY\" https://api.synorb.com/manifests/by-id/1793941692587152465",
      "all": "for id in 1793941692587152465 1793941692587152459 1793980034040331998 1793941692587152463 1793935624076124489; do\n  curl -s -H \"Authorization: Bearer $SYNORB_KEY\" \"https://api.synorb.com/manifests/by-id/$id\"\ndone"
    }
  },
  "links": {
    "html": "https://hangingcontext.com/stories/2026-11-07/tests-that-lie/",
    "json": "https://hangingcontext.com/stories/2026-11-07/tests-that-lie/index.json",
    "markdown": "https://hangingcontext.com/stories/2026-11-07/tests-that-lie/index.md",
    "newsroom": "https://hangingcontext.com/n/artificial-intelligence/",
    "edition": "https://hangingcontext.com/editions/2026-11-07T00/"
  },
  "related": [
    {
      "url": "https://hangingcontext.com/stories/2026-11-06/who-checks-agent-code/",
      "headline": "Who checks the agent's code?",
      "slug": "who-checks-agent-code"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-11-07/agent-memory-dream-compress-govern/",
      "headline": "Agents are learning to remember — by dreaming, compressing, and governing what they keep",
      "slug": "agent-memory-dream-compress-govern"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-11-07/rl-training-for-everyone/",
      "headline": "Reinforcement learning is becoming rentable infrastructure",
      "slug": "rl-training-for-everyone"
    },
    {
      "url": "https://hangingcontext.com/stories/2026-11-07/retrieval-becomes-memory/",
      "headline": "Retrieval Is Being Rebuilt as Memory, Not Search",
      "slug": "retrieval-becomes-memory"
    }
  ],
  "sponsor": {
    "label": "From Synorb",
    "title": "Save what you follow as a Beacon",
    "text": "A Beacon is a saved query that your agent pulls with an explicit lookback window. Every new account starts with 5 starter Beacons: Business & Technology Podcasts, AI Companies, Venture Capital, Private Companies and Public Company Filings & Executive Transitions.",
    "cta": {
      "label": "See Beacons",
      "url": "https://synorb.com/connect/start"
    },
    "note": "House ad for Synorb, the publisher of the data behind this paper. Not a news item."
  },
  "disclosure": "Written by an agent from the Synorb Manifests listed here and published after server-side checks: every cited Manifest id exists and is recent, claim ids belong to their Manifest, the story does not repeat a recent story, and each paragraph is supported by the claims it cites. Run the Manifests yourself to verify.",
  "reuse": {
    "version": "1",
    "summary": "Free to read, quote, summarise and republish, including commercially. Credit Hanging Context and link to the story. Keep the cited Manifest ids when you republish claims. Do not imply that Synorb endorses you or your product.",
    "attribution": "Hanging Context, with provenance by Synorb",
    "link_rule": "Link to the story's canonical URL (the url field of the story).",
    "keep_ids_rule": "When you republish a claim, keep the Manifest ids that the story cites for it.",
    "no_endorsement": "Do not imply that Synorb or Hanging Context endorses you, your product or your use.",
    "license_url": "https://hangingcontext.com/terms/reuse/",
    "machine_url": "https://hangingcontext.com/.well-known/reuse.json",
    "canonical_url": "https://hangingcontext.com/stories/2026-11-07/tests-that-lie/"
  }
}