Physical AI Lab // Episode 2.2 // One-pager

Training an arm with data

LeRobot + real SO-series arm data: turning demonstrations into an ACT policy that grasps

Three key points

Imitation learning: the policy learns actions from human demonstrations instead of hand-written rules; the input is the image plus joint angles, the output is the next action.
ACT (Action Chunking with Transformers) predicts a short chunk of future actions at once; the model has only about 80 million parameters, and about 100,000 training steps take a few hours on one GPU (O, LeRobot docs).
Training uses real-robot pick-and-place demos from an SO-series arm in the svla_so101_pickplace dataset (50 episodes, O); real-robot data needs real-robot evaluation, and generalization to new objects and lighting is still hard.

O	50 episodes	of real-robot SO-series pick-and-place demos in svla_so101_pickplace (50 episodes, 30 fps, two cameras; meta/info.json: robot_type so100_follower, so101 only in the dataset name)
O	6 values	of position make up the state and the action: 5 arm joints + 1 gripper (feature names in the dataset's meta/info.json)
O	about 80 million	parameters in the ACT policy, lightweight and easy to work with (official LeRobot docs)
O	50 demos	are often enough for ACT to reach high success rates (a general statement in the official LeRobot docs, not measured in this video)
O	100,000 steps	of training take a few hours on a single GPU (official LeRobot docs)