Physical AI Lab // Episode 3.6 // One-pager
Reading the frontier: checking the numbers
Use the daily radar to recompute a paper's key numbers
Three key points
- Inflated numbers are mostly not fraud but misaligned definitions: evaluation data leaking into training (π0 fell from ~96% to ~89%, O), peaks taken as the norm, simulation taken as a real robot, reporting only the best run.
- Before reading the frontier, run a checklist: who said it (O/R/M evidence levels, O), what definition, how it was split, whether the numbers add up (libero_10 has 379, not 500, O), peak or norm, big enough sample (N=5, O).
- The output of a check is "number + one original sentence that directly supports it + evidence level + link". Use the radar as the way in, check against the original, don't stop at the title.
Evidence: O official · R research · M media
| O | 96%→89% | the same fine-tuning experiment: about 96% improvement reported early because evaluation and training sets overlapped, down to about 89% once they were strictly separated |
| O | O · R · M | the frontier radar tags every claim with an evidence level (O official first-hand, R paper or preprint, M second-hand media) and keeps vendor self-reports apart from peer review |
| O | 379 episodes | lerobot/libero_10 actually contains 379 episodes, not the 500 the benchmark description might suggest; check numbers against the original metadata |
| O | N=5 | the improvements above are averaged over only the first 5 held-out episodes (DROID has 20, LIBERO 75); a small sample |
| O | ≈90% lower | after the strict split, fine-tuning π0 cut action-prediction MSE by 89.7% on DROID and 88.9% on LIBERO (open-loop, N=5) |
| R | up to 70% / 50% | the paper reports that, on its simulated and real-world evaluation data, its sequential test needed up to 70% less evaluation than standard batch methods and up to 50% less than state-of-the-art sequential tests for binary outcomes; different baselines, both upper bounds |
Further reading
- AWS · Engineering blog on fine-tuning π0 (the 96%→89% split lesson)
https://aws.amazon.com/blogs/physical-ai/fine-tuning-%CF%800-pi-zero-for-robotic-manipulation-on-amazon-sagemaker-hyperpod-eks/ - Physical AI frontier radar (every item tagged O/R/M, with checkable numbers extracted)
https://github.com/noteflowai/physical-ai-radar - π0: A Vision-Language-Action Flow Model (arXiv 2410.24164)
https://arxiv.org/abs/2410.24164