Human-level IQM on Atari 100k within one GPU-hour per run, with interval estimates over 10 seeds
Difficulty 32: human-level IQM is already met at about 6–10 GPU-hours, and BBF's cheaper variant runs near the one-hour budget below IQM 1.0, so what remains is efficiency engineering; level with the GPU-kernel parity targets. Application value 15 and leverage 14: Atari 100k is a narrow, saturating proxy.
The problem
Reach human-level play on 26 Atari games from two hours of experience, using an hour of one graphics card.
- 1Give agent 100k steps per game
- 2Score vs human on 26 games
- 3Average with robust IQM
- 4Count GPU time per run
- 5Check confidence interval
Why it matters. Deep RL agents are notoriously hungry for data and compute. Atari 100k asks for human-level play from only 100,000 actions per game, about two hours of gameplay, on 26 games. That data target has been met, but the best agents still need hours of GPU time per game. Cutting that cost makes the benchmark usable by small groups and is a step toward agents, such as robots, that learn quickly and cheaply.
Path to understanding
foundations → problemThe research entries this problem rests on, in the order to read them. Each links to its explainer.
- Start with
- Step 2
- Policy gradient methods and PPOAlgorithm
- Step 3
- Step 4
- Step 5
- Then
- Human-level IQM on Atari 100k within one GPU-hour per run, with interval estimates over 10 seedsThe problem
State of the art
BBF (Schwarzer et al., 2023, Table A.1) reports IQM 1.045, optimality gap 0.344, median 0.917 and mean 2.247 over 26 games, averaged over 50 seeds per game; the paper gives about 6 hours on a single GPU in one place and 10 hours with a CPU and half an A100 in another. EfficientZero (NeurIPS 2021) reports mean 1.945 and median 1.116 in the BBF table (the preprint's own abstract reports 194.3% and 109.0% under its own protocol) but trains in about 8.5 hours using roughly 512 CPU cores and 4 GPUs. The BBF paper also runs a cheaper replay-ratio-2 variant (it reports 40k steps in one hour on half an A100) whose IQM in its ablations is below 1.0, so the one-GPU-hour regime is close to, but not at, human-level IQM. BBF's authors note that no agent was then human-level on all 26 games. Later model-based agents (EfficientZero V2, DIAMOND and others) use search or large world models and report their own aggregates under differing protocols; none of them reports human-level IQM within one GPU-hour per run.
Realistic applications
Cheaper sample-efficient RL makes agent research accessible to small groups and is a proxy for data-efficient learning in robotics where each step is costly.
What counts as solved
A released codebase that, run by an independent party on stated hardware, meets the IQM and compute thresholds with the confidence-interval condition above.
How it gets verified
ReproductionA training run, experiment or computation rerun by someone else.
Benchmark: ALE with the Atari 100k settings (26 games, 100k agent steps, sticky actions reported separately) and human and random reference scores from the BBF or rliable code. Metric: human-normalised IQM across games with a 95% stratified bootstrap CI from the rliable library; also report optimality gap and median. Threshold: IQM at least 1.0 with the CI lower bound at least 0.9. Seeds: at least 10 per game. Compute: at most 1 A100-hour per game-seed end to end, including any evaluation. Verification: an independent party reruns the released code and reports GPU-hours measured by wall clock.
Claims under verification
+ add a claimNo claimed results on the Verification board for this problem yet.
Where the work is
Mostly systems and algorithm engineering to cut wall-clock per run; a run is cheap but 26 games times 10 seeds is the cost.
Bottlenecks
ranked by votes- #1EngineeringHigh replay ratio cost
Sample efficiency in BBF comes from many gradient updates per environment step; reducing wall-clock requires faster networks and data pipelines.
Community grades
Sign in to add your own grade.