Optimizers, initialisation, normalisation and why gradient descent on huge networks works as well as it does.
Computing exact gradients of any program in roughly the cost of running it once.
A rule for scaling initialization and per-layer learning rates with width so that the best learning rate found on a small model is also near-best on a much larger one, avoiding tuning at full scale.