Improve an explanation

Every page explains its idea for a first-year student first, then gives the Expert text under “Go deeper”. The one-line summary, picture, glossary and reading list help too. Rewrite any part so more people can follow it. A maintainer who didn't write your change reviews it, and approved changes show up live in the change log.

ExplainingOffline reinforcement learningAI-drafted, not yet edited by a person

Plain explanation120–250 words, no formulas. What it is, why anyone cares, one everyday analogy (and where the analogy breaks).

Imagine you want to train an AI to help doctors choose treatments. You cannot let it experiment on patients. But hospitals have years of records: what was done and what happened. Offline reinforcement learning tries to learn a better strategy from such records alone.

This is harder than it sounds. In the records, doctors only tried some options. If the AI is allowed to wonder "what if I did something nobody tried?", it has no data to answer, and it will tend to be over-optimistic about those untried options. It may then choose them and fail.

The main idea to deal with this is caution. Offline methods are built to be pessimistic about anything the data does not cover. They either keep the AI's choices close to what was done in the records, or they lower their estimates for untried actions.

Another headache: how do you know which trained version is best? Normally you would test each one, but that is exactly what you cannot do. Finding ways to judge a strategy using only the old records is an active research problem.

Why this changeOne line, like a commit message.

Student explanation250–500 words at first-year university level. The key definitions and the main idea of how it works.

**What it is for.** Ordinary RL learns by trying things, including bad things. A hospital cannot try random drug doses on patients and a self-driving car cannot crash in order to learn, but both have large logs of past decisions and outcomes. Offline RL learns a policy from such a fixed dataset, with no new interaction allowed.

**Example.** A dataset holds a million recorded steps of a walking robot, collected by an earlier, mediocre controller. Can a better controller be learned from those records alone?

**Why the obvious method fails.** Q-learning estimates $Q(s,a)$, the expected future reward of taking action $a$ in state $s$ (an expected value: a probability-weighted average of outcomes), and then picks the action with the highest $Q$. Its update is
$$Q(s,a)\leftarrow r+\gamma\max_{a'}Q(s',a'),$$
the reward just received plus the discounted value of the best next action. The $\max$ looks at every possible next action, including ones that never appear in the data. For those, the network's estimate is pure extrapolation, and some extrapolated values are too high by chance. The max picks exactly those errors, the policy follows them, and no new data ever corrects them. Fujimoto, Meger and Precup (2018) called this extrapolation error; the general name for the mismatch between the data's actions and the learned policy's actions is distribution shift.

**The fix: pessimism.** Successful methods stay cautious where data is thin.
- *Policy constraints* (BCQ, TD3+BC) keep the new policy close to the actions in the data.
- *Conservative Q-Learning* (CQL, 2020) adds a penalty that pushes down $Q$ for actions outside the data, so the learned value underestimates rather than overestimates.
- *Implicit Q-Learning* (IQL, 2021) never evaluates actions outside the data at all.
- *Decision Transformer* (2021) treats the problem as predicting the next action in a sequence, conditioned on the return wanted.

Theory backs the idea: for small (tabular) problems, a pessimistic model-based method needs the fewest samples possible up to constant factors (Li et al., 2022), provided the data covers the states a good policy visits.

**Benchmarks and the open gap.** D4RL (2020) provides standard datasets for simulated robots and mazes, scored so that 0 is a random policy and 100 an expert. A persistent practical problem is choosing hyperparameters (settings such as learning rate or penalty strength) without testing candidates in the real environment, which defeats the purpose of going offline. That requires off-policy evaluation, estimating how well a policy would do from logged data alone, which has no general guarantee when the policy takes actions the log rarely contains.

GlossaryEvery hard word on the page, one per line: term | plain meaning | link (optional).

reinforcement learning | Learning by trial and error: an agent acts, gets rewards, and adjusts its behaviour to collect more reward over time. | https://en.wikipedia.org/wiki/Reinforcement_learning
policy | The rule an agent uses to choose an action in each situation. In deep RL it is usually a neural network. | https://en.wikipedia.org/wiki/Reinforcement_learning#Policy
reward | A number the environment gives after each action, telling the agent how well it is doing. | https://en.wikipedia.org/wiki/Reinforcement_learning
distribution shift | When the situations a model meets differ from the ones it was trained on. | https://en.wikipedia.org/wiki/Concept_drift
Q-value | An estimate of how much total reward follows from taking a given action in a given situation. | https://en.wikipedia.org/wiki/Q-learning
pessimism | In offline RL, deliberately lowering value estimates where data is scarce. | https://arxiv.org/abs/2005.01643
off-policy evaluation | Estimating how well a policy would do using data collected by a different policy. | https://arxiv.org/abs/2103.16596
D4RL | A standard set of datasets for testing offline RL. | https://arxiv.org/abs/2004.07219
expected value | The probability-weighted average of all possible outcomes. | https://en.wikipedia.org/wiki/Expected_value
hyperparameter | A setting chosen before training, such as the learning rate or penalty strength, rather than learned from data. | https://en.wikipedia.org/wiki/Hyperparameter_(machine_learning)
extrapolation | Predicting values outside the range covered by the data, where errors can be large. | https://en.wikipedia.org/wiki/Extrapolation
discount factor | A number γ slightly below 1 that makes rewards further in the future count less. | https://en.wikipedia.org/wiki/Markov_decision_process

Why this changeOne line, like a commit message.