Physical AI Lab // Episode 1.8 // One-pager

Evaluation: What one success rate hides

Change the conditions, change the seed. Does the success rate still hold?

Three key points

For a success rate, ask under what conditions it was measured: change the light or the camera and the result changes; a "net gain of 2" actually hides 3 better and 1 worse (O, paired comparison).
Success and failure each come in more than one kind: all 14 failures stopped only when the budget ran out, with the end of the arm still moving (44.8–138.6 mm). The "failure" label hides how it actually failed (O).
One result doesn't count: random seeds make the score jump around, so repeat, look at the spread, and report a confidence interval. A 30/30 rerun only shows it reproduces under the same conditions; it's not a general conclusion.

O	30 runs	closed-loop simulation tests, one task, three conditions, ten paired starts (robot-reel stress lab, a small controlled experiment)
O	160 actions / 8 s	fixed budget per test: at most 160 actions and 8 seconds of simulated time (robot-reel stress lab)
O	3 better, 1 worse	behind a moved camera's "net gain of 2" successes: split apart, it is really 3 gains and 1 loss (robot-reel stress lab)
O	14 failures	all 14 stopped only after using up the action budget (robot-reel stress lab)
O	44.8–138.6 mm	failures weren't stuck: in the final tenth, the end effector of the 14 failed runs still moved 44.8–138.6 mm (above the 1 mm stall threshold, robot-reel)
O	30 / 30	matched: the full plan rerun on the same machine, with all 30 outcomes, action counts, states and actions identical (robot-reel stress lab)
R	10 runs	10 runs with the same hyperparameters, changing only the random seed; split into two groups of 5 and averaged, the two groups' conclusions can differ significantly (paper reports)