Physical AI Lab // Episode 3.3 // One-pager
RL post-training: the self-improvement loop
From imitation to trial and error, better with practice
Three key points
- Demonstrations rarely cover mistakes and recoveries, so an imitation policy that drifts outside what the demos showed easily gets stuck (distribution shift); reinforcement learning lets a policy learn recovery from its own experience (including rewards and corrections).
- The paper says RECAP brings RL to VLAs (π*0.6) and, on some of the hardest tasks, more than doubled throughput and roughly halved the failure rate (R); HIL-SERL does RL directly on real robots, and its paper says it reaches near-perfect in 1–2.5 hours, with 2× the success rate on average and 1.8× faster execution compared with imitation learning baselines and prior RL methods (R).
- Self-improvement is a loop (see, think, act, feedback, update); simulation (NVIDIA Isaac Lab, O) allows massively parallel practice, but real-robot RL is slow and expensive, rewards are hard to set, and there's still a sim2real gap.
Evidence: O official · R research · M media
| R | Advantage conditioning | RECAP: brings reinforcement learning to VLAs through advantage conditioning, with data from demonstrations, autonomous collection and human corrections (Physical Intelligence) |
| R | More than double / roughly half | on some of the hardest tasks, RECAP more than doubled throughput and roughly halved the failure rate (paper) |
| R | Laundry · box assembly · coffee | after RECAP training, π*0.6 folds laundry in real homes, reliably assembles boxes, and makes coffee with a professional machine (paper) |
| R | 1 to 2.5 hours | real-world RL: 1 to 2.5 hours of training reaches a near-perfect success rate (paper) |
| R | 2× / 1.8× | compared with imitation learning baselines and prior RL methods, success rates are 2× on average, with execution 1.8× faster (paper) |
| O | GPU-parallel simulation | NVIDIA Isaac Lab: an open-source, GPU-accelerated robot learning framework supporting reinforcement learning, learning from demonstration, and motion planning |
| O | +24.5 pt / +16.61 | This episode's simulation, 3 training seeds, paired: success gain [−9.0, 57.9] percentage points includes 0; return [13.48, 19.74] excludes 0; seed 1 alone +40.0 points |
Further reading
- π*0.6: a VLA that learns from experience (RECAP, arXiv 2511.14759)
https://arxiv.org/abs/2511.14759 - Human-in-the-loop real-world reinforcement learning (HIL-SERL, arXiv 2410.21845)
https://arxiv.org/abs/2410.21845 - NVIDIA Isaac Lab documentation
https://isaac-sim.github.io/IsaacLab/main/index.html