■ Block 01 / research
Research
Foundations other work depends on. Each entry explains what it is, why it is load-bearing, its open problems, and a reviewed timeline of progress.
Foundations other work depends on. Each entry explains what it is, why it is load-bearing, its open problems, and a reviewed timeline of progress.
The attention-based neural network design behind modern language models, and the quadratic cost of attention that drives work on exact, approximate and recurrent alternatives.
Sequence layers that carry a fixed-size recurrent state, giving linear-time training and constant-memory decoding; the main architectural alternative to attention, with a known recall gap.
Sparse transformer layers send each token to a few of many expert networks, decoupling parameter count from per-token compute; the router and its load balancing decide whether the experts are actually used.