Physical AI Lab // Episode 3.6 // One-pager

Reading the frontier: checking the numbers

Use the daily radar to recompute a paper's key numbers

Three key points

  1. Inflated numbers are mostly not fraud but misaligned definitions: evaluation data leaking into training (π0 fell from ~96% to ~89%, O), peaks taken as the norm, simulation taken as a real robot, reporting only the best run.
  2. Before reading the frontier, run a checklist: who said it (O/R/M evidence levels, O), what definition, how it was split, whether the numbers add up (libero_10 has 379, not 500, O), peak or norm, big enough sample (N=5, O).
  3. The output of a check is "number + one original sentence that directly supports it + evidence level + link". Use the radar as the way in, check against the original, don't stop at the title.

Evidence: O official · R research · M media

O96%→89%the same fine-tuning experiment: about 96% improvement reported early because evaluation and training sets overlapped, down to about 89% once they were strictly separated
OO · R · Mthe frontier radar tags every claim with an evidence level (O official first-hand, R paper or preprint, M second-hand media) and keeps vendor self-reports apart from peer review
O379 episodeslerobot/libero_10 actually contains 379 episodes, not the 500 the benchmark description might suggest; check numbers against the original metadata
ON=5the improvements above are averaged over only the first 5 held-out episodes (DROID has 20, LIBERO 75); a small sample
O≈90% lowerafter the strict split, fine-tuning π0 cut action-prediction MSE by 89.7% on DROID and 88.9% on LIBERO (open-loop, N=5)
Rup to 70% / 50%the paper reports that, on its simulated and real-world evaluation data, its sequential test needed up to 70% less evaluation than standard batch methods and up to 50% less than state-of-the-art sequential tests for binary outcomes; different baselines, both upper bounds

Further reading