Close the ProcGen generalisation gap: agents trained on 200 levels matching agents trained on unlimited levels
Difficulty 48: generalisation from limited variation is a core RL weakness and partial methods (augmentation, regularisation, decoupled policy and value) leave game-specific gaps without a principled fix. Application value 30: relevant to sim-to-real transfer only indirectly. Leverage 28: benchmark-specific.
The problem
Train on 200 levels of a game and do as well on brand-new levels as someone trained on endless levels.
- 1Train on 200 fixed levels
- 2Test on unseen levels
- 3Compare with unlimited-level agent
- 4Find where the gap remains
- 5Try better regularisation
Why it matters. An agent that only works in the exact levels it trained on is useless in the real world, where every kitchen or road is slightly different. ProcGen tests this directly: its 16 video games generate their levels by a program, so there is an endless supply, and an agent can be trained on a few levels and tested on new ones.
Path to understanding
foundations → problemThe research entries this problem rests on, in the order to read them. Each links to its explainer.
- Start with
- Step 2
- Policy gradient methods and PPOAlgorithm
- Step 3
- Step 4
- Step 5
- Then
- Close the ProcGen generalisation gap: agents trained on 200 levels matching agents trained on unlimited levelsThe problem
State of the art
The ProcGen paper reports that agents trained on small level sets generalise worse and that larger models improve both sample efficiency and generalisation. Phasic Policy Gradient (2020) improved sample efficiency over PPO there. The size of the remaining gap depends on the method (augmentation, regularisation, representation learning). No maintained leaderboard reports the 200-level versus unlimited-level comparison for all 16 games under one protocol, so the target is defined relative to a reference that the claimant trains under identical settings, and the 5% threshold is a DARC-proposed target.
Realistic applications
Generalisation from limited environment variation is a prerequisite for deploying RL in the real world, where the training set of scenes is finite.
What counts as solved
A released method meeting the threshold over 16 games and 5 seeds, rerun by an independent party.
How it gets verified
ReproductionA training run, experiment or computation rerun by someone else.
Benchmark: ProcGen, 16 games, easy difficulty, 200 training levels, 25M timesteps (the recommended easy-mode protocol). Metric: mean normalised return on the full level distribution, averaged over games, with the same algorithm trained on unlimited levels as the reference. Threshold: test return of the 200-level agent at least 95% of the reference. Seeds: at least 5, reported with bootstrap CIs. Compute: stated; 25M timesteps over 16 games is roughly 400M environment steps per seed.
Claims under verification
+ add a claimNo claimed results on the Verification board for this problem yet.
Where the work is
Needs new regularisation or representation ideas as well as experiments across all 16 games.
Bottlenecks
ranked by votes- #1TheoryWhich invariances to learn
There is no principled account of which augmentations or inductive biases give level-generalisation in all games without game-specific design.
Community grades
Sign in to add your own grade.
Connected problems
- This problem is related to Human-level IQM on Atari 100k within one GPU-hour per run, with interval estimates over 10 seeds
Progress
- Dec 3, 2019ProcGen releasedsource
Benchmark and recommended protocols published.