Physical AI Lab // Episode 2.8 // One-pager

Reproducible evaluation: repeated trials and confidence intervals

How sure can you be about one success rate?

Three key points

  1. A single success rate misleads: change only the random seed and re-evaluate, and the number jumps (in a paper's experiments, conclusions differed significantly, R). Give an interval.
  2. Don't use the Wald (normal-approximation) interval; its coverage is persistently poor (R). For small samples use Wilson (R). Bootstrap resampling assumes no distribution, but it treats the sample as a stand-in for the population and breaks down with very few runs or an extreme rate; it suits larger samples and complex statistics (the paper used ten thousand resamples for intervals on MuJoCo average returns, R).
  3. Interval width shrinks roughly as 1/√n: to halve the uncertainty, you need about four times the samples. Zero failures isn't zero risk; the fewer the samples, the looser the upper bound.

Evidence: O official · R research · M media

R10 runswith the same hyperparameters, changing only the random seed; split into two groups of five and averaged, the conclusions can differ significantly (paper)
R10,000 resamplesused by the paper to compute bootstrap 95% confidence intervals for average returns on several MuJoCo environments (pivotal method)
RWald→Wilsonthe standard Wald (normal-approximation) interval has persistently poor coverage; the paper recommends Wilson for small samples, Agresti–Coull for large ones

Further reading