Physical AI Lab // Episode 1.7 // One-pager
Reinforcement learning Getting better by trial and error
No demonstrations, only rewards. Let it find the way itself
Three key points
- Reinforcement learning runs on rewards and trial and error, not demonstrations: right moves earn points, wrong moves lose them, and the robot works out its own policy (situation → next move).
- Trial and error takes many tries, so it moves into simulation: one paper ran thousands of robot-dog copies in parallel on one graphics card, and they learned to walk on flat ground in under four minutes (R, 2021).
- Rewards are hard to get right: write one a little wrong and the robot finds a loophole, scoring high without really doing the job. So rewards need repeated tuning, and someone has to watch the replays.
Evidence: O official · R research · M media
| R | Thousands | of simulated robots trained in parallel on one graphics card, cutting training to minutes (paper, ANYmal robot dog) |
| R | Under 4 minutes | for a quadruped robot dog to learn to walk on flat ground, trained in parallel in simulation; about 20 minutes on uneven terrain (the paper reports) |
Further reading
- Learning to Walk in Minutes (arXiv 2109.11978, thousands of copies in parallel)
https://arxiv.org/abs/2109.11978 - Quadruped reinforcement learning running on a real robot (arXiv 1901.08652, Science Robotics)
https://arxiv.org/abs/1901.08652 - Concrete Problems in AI Safety (covers reward hacking, arXiv 1606.06565)
https://arxiv.org/abs/1606.06565