Physical AI Lab // Episode 3.3 // Quiz
RL post-training: the self-improvement loop
From imitation to trial and error, better with practice
01
A coffee-pouring policy is trained on many successful demonstrations. In a test its hand shakes, coffee spills, and then it freezes. What is the best explanation?
There aren't enough demos yet; more of the same successful demos will fix it
Imitation learning only works in simulation, so on a real robot it freezes anyway
The policy's inference is too slow to react in the instant of the spill
The demos hardly contain post-spill states, so it's somewhere it hasn't seen
02
Compared with simply collecting more demos and imitating them, what does RECAP (π*0.6) add?
It uses only data the robot collects itself, with no human demos
It trains at scale only in simulation, then moves everything to the real robot
RL via advantage conditioning, mixing in autonomous data and human corrections
It makes the model much bigger, so scale alone teaches it to recover
03
Someone says: "HIL-SERL reached near-perfect success in 1 to 2.5 hours, so real-robot RL takes only a few hours for any robot task." Which point in the video most directly rebuts this?
That number comes from pure simulation, not from a real robot
These results are mostly one skill on one machine, far from general
The paper only reports execution speed, not a change in success rate
Real-robot RL needs no human help at all, so it can't be compared
04
In simulation you train an arm with RL to push a block to a target. Return keeps rising, but the replay shows it just shoves the block against the table edge and never reaches the target. What is the most likely cause?
The reward is mis-set and the policy exploits it with unwanted behaviour
Too few training seeds, so the rising return is a lucky fluctuation
Domain randomisation shakes things too hard, so it can't locate the block
The reward is too sparse, so the policy hasn't learned how to push yet
05
This episode's experiment ran 3 training seeds. The 95% confidence intervals of the paired differences across seeds: return 13.48 to 19.74, success rate −9.0% to 57.9%. By the episode's criterion, what is the conclusion?
Both improved, because the success rate rose in all 3 seeds
Return improved; success includes 0, so evidence is insufficient
Neither counts, because the two groups' own intervals overlap
Success improved, because within each seed the intervals don't overlap
Check