Physical AI Lab // Episode 2.8 // Quiz

Reproducible evaluation: repeated trials and confidence intervals

How sure can you be about one success rate?

01A new and an old policy are each tested 20 times: the new one succeeds 15 times, the old one 13 times, and their Wilson intervals overlap heavily. What can you conclude?

02All 10 trials succeed, and the Wald interval gives 100% for both bounds. Why can't you trust this interval?

03The interval's half-width is now about ±10 percentage points. To shrink it to ±5, roughly how many times as many trials do you need?

04A report says "92% success", but it turns out the authors ran 12 rounds and reported only the best one. What is wrong?

05On the real robot you ran 20 trials with no failures. What is the most honest way to report it?