Transformers, state-space models, mixtures of experts and the building blocks networks are assembled from.
The attention-based neural network design behind modern language models, and the quadratic cost of attention that drives work on exact, approximate and recurrent alternatives.
Sequence layers that carry a fixed-size recurrent state, giving linear-time training and constant-memory decoding; the main architectural alternative to attention, with a known recall gap.