ViBench asks agents to build whole products from scratch, and the failures cluster where features meet

A different kind of test

Replit released ViBench 2.0, a benchmark for long-horizon software development, and its defining choice is the starting point. ViBench 2.0 requires agents to build products from an empty project rather than working within an existing codebase. That single decision changes what is being measured. Fixing a bug in mature code tests comprehension. Building from nothing tests planning, sequencing and the ability to keep earlier decisions consistent as requirements pile up. m:…9231

The scale of the test is large enough to make the results hard to dismiss as a toy. ViBench 2.0 includes 17 products, 241 feature requests and 544 test plans. It is also part of the Specialized Intelligence Index from Fireworks. m:…9231

GPT-6.1 Sol leads ViBench 2.0 results with a pass@1 of 65%. That is the headline number, and it should be read both ways. A leading model passes roughly two thirds of builds on the first try, which is genuinely useful for prototyping. It also fails roughly a third, which means any team treating the output as finished software is skipping the step the benchmark itself is designed to expose. m:…9231

Where the builds break

Replit says top models frequently fail when features intersect or when products require maintaining state across evolving requirements. That failure mode is familiar to anyone who has managed a real codebase. Individual features work in isolation and then collide at the seams, or an early assumption quietly contradicts a later requirement. Long-horizon tests catch this precisely because they refuse to reset the board between tasks. m:…9231

A preprint on evolving features tells the same story from a different angle. Coding agent success rates drop from 62.7% at the first stage to 28.5% when the whole feature is tested on the final code. The collapse between first stage and final code is the number to remember. It suggests that early progress in agent-built software is a weak predictor of a finished feature, and that demos showing a first stage working are not evidence about the last stage. m:…0491

The same preprint found inherited defects are a common failure mechanism, present in 72.9% of failed later stages in multi-stage coding tasks, and that in stages following a failure, agents leave the earlier error unnoticed about 95% of the time. That is a compounding problem, not a collection of independent mistakes. An error that survives one stage becomes the foundation of the next, and the agent rarely goes back to question it. Teams using agents for multi-stage work need review gates between stages, because the agent itself is unlikely to supply them. m:…0491

Speed, cost and the tradeoff nobody escapes

ViBench 2.0 also publishes the tradeoff that engineering managers actually have to choose between. Opus 5.5 finishes a build in about 70 minutes on average, roughly 2.5 times faster than GPT-6.1 Sol. Speed at that scale changes how many iterations a team can afford in a day, and it changes the feedback loop for the humans reviewing the work. m:…9231

Cost points the other way. GPT-6.1 Sol costs about $12 per build, while GPT-6.1 Terra passes 47% of builds at about half the cost of GPT-6.1 Sol, and DeepSeek V4.1 Flash passes 77% of its test plans for about $2.50 per build. Meanwhile GPT-6.1 Astra costs about $90 per build. These are Replit-reported figures for its own benchmark, so they should be treated as directional rather than as a price list. Even directionally, they describe three different products: the fastest build, the cheapest acceptable build, and the highest-scoring build, and no single model is all three. m:…9231

That tradeoff is exactly why benchmark choice matters, and a second preprint warns against taking benchmark labels at face value. Benchmark labels such as bug fixing and feature implementation are insufficient for determining whether evaluated work resembles a team's actual workload. Every pair of benchmarks analyzed differs on at least two SNC axes, including benchmarks with the same task label. In plain terms, two tests both called feature implementation can measure substantially different work, so a leaderboard ranking is only meaningful if the test resembles your work. m:…6974

The same preprint notes that resolved agent runs tend to involve less distributed changes and code with lower values on the Centrality indicators. That is a quiet warning about selection effects. Agents look best on work that is already contained. The work that sprawls across a codebase, which is often the work that matters most, is where they struggle. ViBench earns its keep by forcing the sprawl back into the test. m:…6974

What does not help

One popular remedy does not survive measurement. A preprint on agent skills found that 92 of 98 skills have no statistically significant effect on agent performance, and adding skills only increases average pass ratio from 76.6% to 76.9%. Worse, harmful skills can cause performance drops of up to 17.0 points by overriding task instructions. The implication is blunt: a library of skills is not a substitute for a better agent or a better review process, and a bad skill actively hurts. m:…4885

Taken together, the evidence points to a practical discipline for teams adopting coding agents. Test on whole products built from scratch, not just on isolated tasks. Review between stages, because errors compound and agents rarely revisit them. Choose the model by the tradeoff your project actually needs, speed, cost or pass rate, rather than by a single headline score. m:…9231 m:…0491 m:…4885