Physical AI Lab // Episode 2.8 // Quiz
Reproducible evaluation: repeated trials and confidence intervals
How sure can you be about one success rate?
01
A new and an old policy are each tested 20 times: the new one succeeds 15 times, the old one 13 times, and their Wilson intervals overlap heavily. What can you conclude?
The new one is better, by 10 percentage points
The old one is better, because its results are more stable
No winner yet; more trials are needed
The two policies have exactly the same true success rate
02
All 10 trials succeed, and the Wald interval gives 100% for both bounds. Why can't you trust this interval?
Wald only works for continuous data, not success rates
Observing 100% must be a calculation error
Wald only suits cases with many failures
The normal approximation breaks down here
03
The interval's half-width is now about ±10 percentage points. To shrink it to ±5, roughly how many times as many trials do you need?
Twice as many
Four times as many
One and a half times as many
Ten times as many
04
A report says "92% success", but it turns out the authors ran 12 rounds and reported only the best one. What is wrong?
Reporting only the best round inflates the number
12 rounds is too few; you need at least 100
They should report the worst round to be conservative
92% is above the Wilson interval's upper bound
05
On the real robot you ran 20 trials with no failures. What is the most honest way to report it?
100% success; this policy doesn't fail
Zero failures in 20, with a confidence interval
Don't report yet; wait for the first failure
Just write "no failures observed", with no interval
Check