Physical AI Lab // Episode 2.8 // One-pager
Reproducible evaluation: repeated trials and confidence intervals
How sure can you be about one success rate?
Three key points
- A single success rate misleads: change only the random seed and re-evaluate, and the number jumps (in a paper's experiments, conclusions differed significantly, R). Give an interval.
- Don't use the Wald (normal-approximation) interval; its coverage is persistently poor (R). For small samples use Wilson (R). Bootstrap resampling assumes no distribution, but it treats the sample as a stand-in for the population and breaks down with very few runs or an extreme rate; it suits larger samples and complex statistics (the paper used ten thousand resamples for intervals on MuJoCo average returns, R).
- Interval width shrinks roughly as 1/√n: to halve the uncertainty, you need about four times the samples. Zero failures isn't zero risk; the fewer the samples, the looser the upper bound.
Evidence: O official · R research · M media
| R | 10 runs | with the same hyperparameters, changing only the random seed; split into two groups of five and averaged, the conclusions can differ significantly (paper) |
| R | 10,000 resamples | used by the paper to compute bootstrap 95% confidence intervals for average returns on several MuJoCo environments (pivotal method) |
| R | Wald→Wilson | the standard Wald (normal-approximation) interval has persistently poor coverage; the paper recommends Wilson for small samples, Agresti–Coull for large ones |
Further reading
- Brown, Cai & DasGupta · Interval Estimation for a Binomial Proportion (2001, Statistical Science)
https://projecteuclid.org/journals/statistical-science/volume-16/issue-2/Interval-Estimation-for-a-Binomial-Proportion/10.1214/ss/1009213286.full - Henderson et al. · Deep Reinforcement Learning that Matters (arXiv 1709.06560)
https://arxiv.org/abs/1709.06560