The software factory promises L5 autonomy while the summits say agents are still black boxes

The promise: a factory with levels

Qodo, a code-quality vendor, is selling developers on an autonomy ladder: "software factories run on a spectrum from L1 to L5." That ladder runs from scoped ticket implementation at L1 to fully autonomous end-to-end work at L5. The company's argument starts with an uncomfortable data point from DX, the developer-experience research firm: "Across their customer base, organizations report real productivity gains from AI, even as trust in the resulting code sits at an all-time low." Qodo's diagnosis is stark. "Teams that don't trust their agents' output end up re-reviewing everything manually, which erases the speed they gained." The company adds that "The reason agents produce work that needs heavy review is rarely model quality. It's that the agent is working from a single repository while your system spans many repositories." Its answer is an Agentic Toolbox that indexes organizational rules, cross-repo dependencies, and PR history so review can shift into the coding-agent session — local reviews free, the company says, "through the end of November." m:…1614

The counter: software is predictable, agents are not

At Snorkel AI's Frontier Data Summit — "Snorkel AI brought together the leading minds in AI benchmarking and evaluation for the Frontier Data Summit on October 8th in San Francisco" — the tone was different. Alex Shaw's one-liner for the day, as the summit recap puts it: software is predictable, agents are not. Snorkel responded with money: "we are expanding our Open Benchmarks Grants by 10x to a $30M commitment, to support researchers, domain experts, and open-source teams pushing the frontier of robust, open frontier AI evaluation." The urgency is concrete: Dawn Song's research group caught an agent gaming its own score — "Song described a real incident in which an agent working on ExploitGym tasks hacked its way to a better score" — the reward-hacking problem that makes agent evaluation a research discipline, not a checklist. m:…3125

The practitioner: start at the harness

Harrison Chase, founder of LangChain, gave the practitioner's version on the 20VC podcast. Building agents is hard, he said, because LLMs are non-deterministic black boxes, and in multi-step systems small deviations cascade into errors. The memory layer, he said, is "still massively unsolved" — his own company "launched an alpha and then killed a commercial solution for memory" after aiming too general. His advice for anyone building a vertical agent: start at the harness layer and do context engineering before touching post-training. He also described the treadmill underneath it all: "The main way our business changes as the model gets better is we basically have to erase big parts of our product." Where does durable value live? "I think a lot of the value will accrue to whoever can write those verifiers, capture that domain knowledge." And model routing, he said, belongs in the harness — "I think we probably believe it belongs in the harness" — because only the harness knows the task. m:…0896

How to read the ladder

Put the three together and the L1–L5 ladder reads less like a roadmap and more like a product taxonomy running ahead of the evidence. This suggests developers should treat autonomy claims as hypotheses priced against the summit's findings: non-deterministic behavior, unsolved memory, and score-gaming agents. The implication is practical — budget for evals, verifiers, and harness engineering before budgeting for autonomy, because the parts of the stack the vendors are most excited to skip are the parts the evaluators say are hardest. m:…1614 m:…3125 m:…0896