Reverse-mode automatic differentiation
Computing exact gradients of any program in roughly the cost of running it once.
Every gradient-based training loop in modern ML, plus large parts of scientific optimization and differentiable simulation.
A trick that shows how every knob in a huge program affects its result, for about one run's cost.
- 1Model has billions of knobs
- 2Nudging each knob is too slow
- 3Run once, record every step
- 4Walk backwards applying chain rule
- 5All gradients for ~one run's cost
The setting. A program computes a number (say, a training loss) from many inputs (the parameters of a model). To improve the model, gradient descent needs the gradient: the list of how much changes per small change in each . Writing the formula for it by hand is error-prone, and estimating it by nudging each input separately costs extra runs. For in the billions that is impossible.
Go deeper
The full technical text.
What it is
A program is a composition of primitive operations. Reverse-mode AD records those operations on a tape during the forward pass, then walks the tape backwards, applying the chain rule to accumulate the derivative of a single scalar output with respect to every input.
Why it matters
For a function the full gradient costs a small constant multiple of evaluating , independent of . That is what makes training networks with billions of parameters possible at all.
Words on this page (11)
- gradient
- The list of numbers saying how much an output changes when each input is nudged slightly; it points in the direction of fastest increase.
- derivative
- How fast one quantity changes when another changes by a tiny amount; the slope of a curve at a point.
- chain rule
- The rule for the derivative of a chain of steps: multiply the local slopes of each step along the way.
- gradient descent
- Improving a model by repeatedly taking a small step in the direction that lowers the loss, using the gradient to choose the direction.
- backpropagation
- The name for reverse-mode autodiff when it is used to compute gradients in neural networks.
- automatic differentiation
- A family of methods that compute exact derivatives of a computer program mechanically, without symbolic algebra or finite-difference guessing.
- loss function
- A single number scoring how badly a model is doing; training tries to make it small.
- neural network
- A model built from many simple adjustable units, whose settings are tuned on data.
Open problems & questions
No questions tracked yet.
Progress timeline
No reviewed progress yet.