Physical AI Lab // Episode 3.6 // Quiz
Reading the frontier: checking the numbers
Use the daily radar to recompute a paper's key numbers
01
A report says fine-tuning gave "about a 96% improvement". Checking it, you find some of the evaluation episodes also appear in the training set. What is the most reasonable judgement?
The overlap is small, so its effect is negligible and 96% still stands
The authors faked it, so the whole report is untrustworthy and should be dropped
An open-book exam inflates the score; look for the strictly split number
Retrain with more GPUs and the 96% will reproduce reliably
02
The same "90% success rate" appears in a vendor press release, an arXiv preprint and a tech-media article. Using the radar's evidence levels, which reading is right?
All three say 90%, they confirm each other, so treat it as settled
The media article came out last, so the M-level number is the most reliable
O, R and M weigh differently; check the measure in the first-hand source
Levels are quality scores, so an O-level number is always more accurate than R
03
You wrote a training script assuming "this dataset has 500 episodes", and halfway through it fails with an index-out-of-range error. What should you do first?
Count the actual episodes in the original metadata
Lower the batch size to ease memory pressure and rerun
Switch to a bigger dataset so you don't run short of samples
Skip the failing episodes and pad back up to 500
04
In a report, two methods' rankings differ only slightly, and the numbers are averaged over just the first 5 held-out episodes. How should you put it in your check notes?
Write "method A is significantly better than method B"
The sample is too small; the slight gap is within noise
With only 5 held-out episodes, the whole result should be thrown out
Recompute on 20 episodes from the training set so the sample is big enough
05
When checking "action error fell by about ninety percent", which record is the most acceptable?
The number "about 90%", plus the paper's title and link
A summary in your own words of the paper's main contributions
The abstract's line "significantly improves performance", plus the link
The number, a supporting quote, its conditions, level and link
Check