Selecting the best offline-RL policy on D4RL with no environment interaction
Difficulty 50: entangled with off-policy evaluation, which has no general guarantee without coverage assumptions, so a method that works across algorithms and domains is research-grade. Application value 48: needed wherever offline RL is deployed without online testing. Leverage 36: reshapes how offline RL results are reported.
The problem
Pick the best of many trained AI strategies without ever trying them out in the real world.
- 1Train many candidate policies
- 2Score them using logs only
- 3Pick the highest-scoring policy
- 4Check by real rollouts
- 5Measure the regret
Why it matters. Offline RL learns a policy from logged data because testing in the real world is unsafe or costly. But every offline algorithm has hyperparameters, settings such as the learning rate or how strongly to penalise unfamiliar actions, and results swing widely with them. The usual fix, running each candidate policy in the environment and keeping the best, is exactly what offline RL is not allowed to do. Without a way to choose offline, offline RL is not really offline.
Path to understanding
foundations → problemThe research entries this problem rests on, in the order to read them. Each links to its explainer.
- Start with
- Step 2
- Policy gradient methods and PPOAlgorithm
- Step 3
- Offline reinforcement learningAlgorithm · host
- Then
- Selecting the best offline-RL policy on D4RL with no environment interactionThe problem
State of the art
Paine et al. (2020) report that, with the offline RL algorithm and Q-value estimator controlled, candidate policies can be ranked well enough to choose close to the best, but also that offline RL algorithms are not robust to hyperparameter choices. The DOPE benchmark (Fu et al., 2021) shows off-policy evaluation methods vary widely across tasks. A method that works across algorithms and across D4RL domains without tuning on the evaluation environment is not established, and no shared leaderboard records selection regret on D4RL; the regret and correlation thresholds are DARC-proposed targets.
Realistic applications
Offline RL is deployed where online testing is unsafe or costly; without reliable offline selection, practitioners must still run candidates online.
What counts as solved
A method fixed in advance meets the regret and correlation thresholds on at least 9 D4RL tasks with 5 seeds, evaluated by an independent party who holds the ground-truth rollouts.
How it gets verified
ReproductionA training run, experiment or computation rerun by someone else.
Benchmark: D4RL (halfcheetah, hopper, walker2d at medium, medium-replay, medium-expert; AntMaze umaze, medium, large). Candidates: policies from CQL, IQL, TD3+BC and one other algorithm (CORL implementations) over a stated hyperparameter grid and 5 training seeds, with ground truth from at least 100 rollouts per candidate. Metric: regret of the selected policy in D4RL normalised score and Spearman correlation between method scores and true returns. Threshold: mean regret at most 5 points and mean correlation at least 0.7 over at least 9 tasks, with a selection rule fixed before looking at ground truth.
Claims under verification
+ add a claimNo claimed results on the Verification board for this problem yet.
Where the work is
Statistical methodology (off-policy evaluation, model selection) matters most, then benchmark engineering.
Bottlenecks
ranked by votes- #1TheoryOff-policy evaluation under distribution shift
Estimating the return of a policy that visits actions absent from the data has no general guarantees without strong coverage assumptions.
Community grades
Sign in to add your own grade.
Connected problems
No connections yet.