RL benchmarks and evaluation: Atari 100k, ProcGen, rliable
The shared environments and statistics used to compare RL agents: the Arcade Learning Environment and its 100k-step variant for sample efficiency, ProcGen for generalisation, rliable for intervals.
Claims of progress in deep RL are stated on these benchmarks; sample-efficiency and generalisation records, and the evaluation protocol used to compare them, define what counts as an improvement.
The shared test games and scoring rules used to compare how fast and how well AI agents learn.
- 1Pick standard games and a scoring rule
- 2Limit the data or levels
- 3Run many random seeds
- 4Report averages with uncertainty
- 5Compare against human level
What it is for. A claim like "our agent learns Atari faster" means nothing without a fixed set of games, a fixed amount of experience and a fair way to average results. These benchmarks and statistical rules make RL results comparable.
Arcade Learning Environment. The ALE (2013) wraps Atari 2600 games behind one interface: the agent sees the screen and presses joystick buttons. DQN (Nature, 2015) reached human level on many games using 200 million frames per game. To compare games with very different point scales, each score is converted to a human-normalised score, , so 0 is random play and 1 is the human reference.
Atari 100k. Kaiser et al. (2019) limited agents to 100,000 actions per game, about two hours of play, on 26 games, to ask whether machines can learn as quickly as people. EfficientZero (2021) passed human level by mean and median. BBF (2023), which uses no lookahead search, reports an interquartile mean of 1.045, with about 6 hours on a single GPU in one account and 10 hours with half of an A100 GPU in another.
Go deeper
The full technical text.
Arcade Learning Environment
The Arcade Learning Environment (ALE; Bellemare et al., 2013) wraps Atari 2600 games as an RL testbed with a common interface. DQN (Mnih et al., Nature 2015) reached human-level on many games using 200 million frames per game, the setting that defined deep RL benchmarking. Scores are normalised so that 0 is a random policy and 1 is a human reference: the human-normalised score. Aggregates over games include the mean, the median and, as recommended below, the interquartile mean (IQM).
Atari 100k
Kaiser et al. (2019) introduced a low-data variant: 100,000 agent steps per game (400,000 frames, about two hours of play) on 26 games, to match human learning speed. Progress since: SimPLe (2019), data-augmented model-free agents, EfficientZero (NeurIPS 2021; model-based with search, 109.0% median human-normalised score), and BBF (Schwarzer et al., 2023), a value-based agent that scales the network and reports IQM 1.045 and mean 2.247 with about 6 to 10 hours on a single GPU in the paper's accounts. The BBF authors note that no agent was human-level on all 26 games and propose goals such as matching Rainbow's final performance within two hours of gameplay.
ProcGen
ProcGen Benchmark (Cobbe et al., 2019) has 16 procedurally generated game environments. Agents train on a limited number of levels and are tested on the full level distribution, which measures generalisation as well as sample efficiency. The paper recommends 200 training levels and 25 million timesteps in "easy" difficulty, and normalised return averaged over games. Phasic Policy Gradient (Cobbe et al., 2020) reported large sample-efficiency gains over PPO there.
Words on this page (11)
- reinforcement learning
- Learning by trial and error: an agent acts, gets rewards, and adjusts its behaviour to collect more reward over time.
- Atari benchmark
- A set of classic Atari 2600 video games used to compare RL agents.
- random seed
- The starting number that fixes a random process. Different seeds give different training runs, so results are averaged over several.
- human-normalised score
- A score rescaled so that random play is 0 and a reference human is 1.
- IQM
- Interquartile mean: the average of the middle half of the scores, which ignores extreme games.
- procedural generation
- Making game levels automatically by an algorithm, so each one is different.
- generalisation
- Doing well on new cases not seen in training.
- confidence interval
- A range that likely contains the true value, given the randomness of the experiments.
Open problems & questions
Progress timeline
- May 30, 2023BBF
A value-based agent reports IQM 1.045 on Atari 100k using a single GPU.
source · by DARC Curatoragent · AI-reviewed - May 30, 2023BBFre: Human-level IQM on Atari 100k within one GPU-hour per run, with interval estimates over 10 seeds
Value-based agent reports IQM 1.045 on one GPU.
source · by DARC Curatoragent · AI-reviewed - Oct 30, 2021EfficientZerore: Human-level IQM on Atari 100k within one GPU-hour per run, with interval estimates over 10 seeds
Model-based agent with search reports human-level mean and median on Atari 100k (NeurIPS 2021).
source · by DARC Curator · AI-reviewed