Physical AI Lab // Episode 1.5 // One-pager
Where does data come from?
Teleoperation, human video, synthesis: three sources
Three key points
- Teleoperation: a person drives the robot through demonstrations (often with a leader and a follower arm). The motion is the cleanest, but it's slow and expensive. The SO-101 pick-and-place dataset has 50 episodes, each with 2 cameras at 30 frames a second (O); across institutions, data can be pooled into large datasets (Open X-Embodiment, 22 kinds of robot, 527 skills, R).
- Human video: watch people work directly, for far more volume (EgoScale used 20,854 hours of egocentric video, R). But human hands aren't robot hands (the embodiment gap), and the video still needs aligning and labelling.
- Synthesis: multiply a few real demonstrations into huge amounts in simulation (MimicGen, about 200 → over 50,000, 18 tasks, R). Flexible and cheap, but it must stay true to life. Data needs variety, not just volume.
Evidence: O official · R research · M media
| O | 50 episodes | in the SO-101 pick-and-place teleoperation dataset, each with 2 cameras at 30 frames a second, 11,939 frames in all (LeRobot) |
| R | 22 kinds | of robot, their data pooled into one dataset (21 institutions, 527 skills) |
| R | 20,854 hours | of action-labelled egocentric human video used to pre-train robots (the paper says over 20× larger than prior efforts, about 2.4 years non-stop) |
| R | Over 50,000 | demonstrations generated automatically from about 200 human demonstrations (18 tasks, in simulation, paper) |
Further reading
- LeRobot · SO-101 pick-and-place dataset (svla_so101_pickplace)
https://huggingface.co/datasets/lerobot/svla_so101_pickplace - Open X-Embodiment (arXiv 2310.08864)
https://arxiv.org/abs/2310.08864 - EgoScale: large-scale egocentric human data (arXiv 2602.16710)
https://arxiv.org/abs/2602.16710 - MimicGen: multiplying data from a few human demonstrations (arXiv 2310.17596)
https://arxiv.org/abs/2310.17596