Physical AI Lab // Episode 1.8 // Quiz
Evaluation: What one success rate hides
Change the conditions, change the seed. Does the success rate still hold?
01
A report says only: “Our policy has a high success rate.” Which piece of information would help most in judging whether that number holds up?
Under what conditions, and how many trials
The robot's design and how well-known the brand is
Whether the demo video is sharp and smooth
Exactly how high the success-rate figure is
02
After the camera was moved, successes showed a “net gain of two”. To find out whether some starting states actually got worse, what should you look at?
The overall success rate for each of the three conditions
How much of the action budget each trial used
Whether a repeat run gives the same results
Paired results for each starting state
03
All 14 failures used up the 160-action budget, and the end effector was still moving in the final stretch. What is the most reasonable conclusion?
The robot got stuck and stopped dead at the end
It had already failed on its very first action
It didn't give up; it just ran out of budget
All of these failures were caused by the dim lighting
04
You ran a reinforcement-learning method with just one seed, and its success rate beat the baseline. What is the soundest next step?
Write it up straight away: the new method is better
Run several more seeds and report the range
Run one other task; if it matches, it counts
Re-run the same seed exactly to confirm it
05
The experiment was re-run exactly as before, and all 30 results matched. What can you conclude from this?
The policy is reliable on any task at all
Reproducible here; other tasks or seeds may differ
It is an official benchmark, comparable with others
The success rate has been proven to be 100%
Check