Robot benchmarks are measuring the wrong thing

Rankings that do not transfer

RoboColosseum, described in a preprint, is a distributed framework for evaluating robot policies across varied hardware and environments. It was built to address a basic reproducibility problem: physical trials are hard to repeat, so results from one lab setup often say little about another. The study found that current policies lack consistent performance across different embodiments, meaning a policy that works on one robot body may not work on another. m:…4626

The more consequential finding concerns rankings. Zero-shot rankings turned out to be poor predictors of fine-tuned success, so a leaderboard that ranks policies before task-specific training does not reliably tell builders which policy will win after training. The researchers also found that measuring partial progress reveals capability differences that binary success metrics hide: a policy that gets most of a task done looks identical to one that fails immediately when the only score is pass or fail. m:…4626

The partial-progress finding deserves emphasis because it changes what a benchmark can teach. A binary score treats a policy that nearly completes a task as equal to one that fails at the first step, which hides exactly the information a builder needs to decide what to fix. Measuring partial progress turns evaluation from a verdict into a diagnosis, and that is what makes RoboColosseum's distributed framework more than a bigger leaderboard. m:…4626

The generalist gap

A second preprint tested whether frontier vision-language models are ready to act as robot generalists. Researchers built the Embodied Agent Arena, a benchmark of 1,000 cases across five capability domains, and evaluated ten frontier models on it. Closed-weight models consistently outperformed open-weight counterparts in task planning and manipulation. m:…8965

Open-weight models showed lower planning and manipulation success than closed-weight models, and agents could achieve early subgoals but fail to complete the full task sequence, which limited their effectiveness as generalists. The split is instructive: understanding a scene and physically acting in it are different skills, and current open models are further along on the first than the second. m:…8965

Picking a model without trusting the scoreboard

A practitioner comparison of vision-language-action models reaches a compatible conclusion from the builder side. Choosing a model means balancing GPU memory capacity, weight availability, and alignment with specific training data. Smaller models such as SmolVLA are optimized for single-GPU consumer hardware, while OpenVLA requires more than 24 GB of VRAM for fine-tuning, and SmolVLA is the recommended model for single-GPU setups. m:…8741

The comparison warns that LIBERO benchmark scores for VLA models are not comparable across different papers, and notes that OpenVLA's authors dropped 307 of 2,000 LIBERO demonstrations due to failures. Its advice is practical: visually inspect training demonstrations before fine-tuning, since fine-tuning failures in VLA models are primarily caused by data-centric issues. In other words, trust your own data checks more than a published score. m:…8741

Learning from human video instead

One reason evaluation is so hard is that robot data is expensive to collect, which is what makes a fourth preprint notable. RoboTryOn synthesizes robot demonstration videos and joint actions from egocentric human footage, bypassing costly teleoperation. It translates human demonstration videos into robot videos, using an inverse dynamics model with depth features to infer action labels, so policies can learn complex tasks from synthetic data. m:…6293

In experiments, the approach improved success rates on novel bimanual manipulation tasks from 2% to 60% while matching teleoperated demonstration performance at approximately a third of the collection cost. If synthetic demonstrations generalize, the bottleneck for robot learning shifts from collecting teleoperation hours to generating and verifying synthetic ones, and evaluation will need to keep up with that shift too. m:…6293