Coding agents write tests that pass and lie. Proving the fix is the job.

Executable is not faithful

In a preprint, researchers introduce MockBench, a dataset of 165 real-world Java and Python tasks for evaluating how well language models and coding agents generate mock objects, stubs, and assertions for unit testing. Coding agents improve mock artifact executability compared with base language models. That sounds like progress, and in one sense it is: the code runs. m:…2465

The problem is what running proves. The paper reports that executability gains do not consistently translate to higher behavior fidelity or better interaction-fault detection. The generated mocks frequently fail to simulate dependency behavior accurately, so they are executable but unfaithful. The researchers identified ten distinct error types in generated mock artifacts that can conceal behavioral deviations, after analyzing 660 failures. m:…2465

The implication is direct for anyone wiring agents into continuous integration. A test suite that passes because its mocks are wrong is worse than a suite that fails loudly: it certifies behavior that does not exist. The paper's message is that developers should prioritize behavioral validation over simple executability when integrating agentic testing tools. m:…2465

Long horizons break agents

A second preprint tests whether coding agents can keep building on their own code over long horizons. Its Multi-step SWE benchmark contains 69 complex engineering tasks requiring agents to build on their own previous outputs across 5 to 15 dependent steps. Current frontier models achieve a maximum solve rate of 18.5%, and 38 of the 69 tasks were not solved by any of the nine tested frontier models. m:…2459

The mechanism is compounding error. Early design decisions and errors propagate across the dependent steps, so a mistake in step two can doom step twelve. That is the normal shape of real software work, iterative, stateful, and dependent on earlier choices, which is why the benchmark matters more than single-turn coding tests. An agent that writes a good first function and a bad second one is not a junior developer; it is a liability that compounds. m:…2459

Proving a fix is the real work

The security world has already learned this lesson the expensive way. Pixee's guide to pen test remediation argues that a ticket marked done is insufficient evidence of remediation, and that a finding should only be closed when it includes a merged code change, a passing regression test, and a passing retest. Regression tests should be added in the same pull request as the code fix, and retest terms should be agreed when signing the statement of work, before the report lands. m:…1998

Cobalt's 2026 report, cited by Pixee, analyzed more than 16,500 pentests across nearly 3,000 organizations. The top 10% of organizations remediate half of their high-risk findings in 10 days; the bottom 10% take 249 days. Organizations running security testing as a program resolve 4.5 times more critical findings in under three days than compliance-driven teams. The gap is not tooling alone; it is whether remediation is treated as a program or a compliance task. m:…1998

The same verification gap shows up in automated submissions. Google paused rewards for product bugs in its open source bounty program in October 2026 due to a rise in invalid automated submissions. Pixee also notes that most remediation tools do not describe taking a human-written pen test report through to a merged code change, so the last mile is still manual, and it is where fixes are proven or not. m:…1998

Deterministic tooling as the counterweight

A third preprint offers one architectural answer: stop prompting agents and start equipping them. Skillware replaces prompt-based tool use with modular, versioned software bundles containing typed JSON contracts, Python code, and unit tests. The paper reports token usage reduced by up to 84.8% and inference costs by up to 97.3% compared with prompt-based agent tool use, and says small local models of 3 to 8 billion parameters equipped with Skillware outperform unequipped frontier models on complex multi-step workflows. It also claims Skillware eliminates parameter errors in agentic tool execution. m:…2463

The Trelium blog makes the same point without a benchmark: model output can vary from run to run, so reliability usually comes from the system around the model. A production workflow needs a verified business outcome, durable progress tracking, integration access, exception handling, and operator recovery, and progress should be recorded outside the conversation so work can resume from a known point. Partly automated workflows should be evaluated honestly and not described as end-to-end. m:…4489

Read together, the five sources converge on one discipline: never trust the artifact, trust the check. Executable mocks need behavioral validation. Multi-step agents need their intermediate steps verified. Pen test findings need a merged fix, a regression test, and a retest. And agentic workflows need deterministic tooling and durable state, because the model itself will vary. The bottleneck in agentic coding is no longer generating code, it is proving the code does what it claims. m:…2465