Physical AI Lab // Episode 3.3 // One-pager

RL post-training: the self-improvement loop

From imitation to trial and error, better with practice

Three key points

  1. Demonstrations rarely cover mistakes and recoveries, so an imitation policy that drifts outside what the demos showed easily gets stuck (distribution shift); reinforcement learning lets a policy learn recovery from its own experience (including rewards and corrections).
  2. The paper says RECAP brings RL to VLAs (π*0.6) and, on some of the hardest tasks, more than doubled throughput and roughly halved the failure rate (R); HIL-SERL does RL directly on real robots, and its paper says it reaches near-perfect in 1–2.5 hours, with 2× the success rate on average and 1.8× faster execution compared with imitation learning baselines and prior RL methods (R).
  3. Self-improvement is a loop (see, think, act, feedback, update); simulation (NVIDIA Isaac Lab, O) allows massively parallel practice, but real-robot RL is slow and expensive, rewards are hard to set, and there's still a sim2real gap.

Evidence: O official · R research · M media

RAdvantage conditioningRECAP: brings reinforcement learning to VLAs through advantage conditioning, with data from demonstrations, autonomous collection and human corrections (Physical Intelligence)
RMore than double / roughly halfon some of the hardest tasks, RECAP more than doubled throughput and roughly halved the failure rate (paper)
RLaundry · box assembly · coffeeafter RECAP training, π*0.6 folds laundry in real homes, reliably assembles boxes, and makes coffee with a professional machine (paper)
R1 to 2.5 hoursreal-world RL: 1 to 2.5 hours of training reaches a near-perfect success rate (paper)
R2× / 1.8×compared with imitation learning baselines and prior RL methods, success rates are 2× on average, with execution 1.8× faster (paper)
OGPU-parallel simulationNVIDIA Isaac Lab: an open-source, GPU-accelerated robot learning framework supporting reinforcement learning, learning from demonstration, and motion planning
O+24.5 pt / +16.61This episode's simulation, 3 training seeds, paired: success gain [−9.0, 57.9] percentage points includes 0; return [13.48, 19.74] excludes 0; seed 1 alone +40.0 points

Further reading