The transformer architecture
The attention-based neural network design behind modern language models, and the quadratic cost of attention that drives work on exact, approximate and recurrent alternatives.
Nearly all large language models and many vision and protein models are transformers, so its compute and memory costs set the economics of training and inference.
The neural network design behind chatbots, which lets every word look at every other word, at a rising cost.
- 1Words need context from far away
- 2Attention scores every pair of words
- 3Blend information using those scores
- 4Runs in parallel, so training scales
- 5Pair scores make cost grow with n²
Architecture. A transformer maps a sequence of token vectors (each of size ) to a sequence of the same shape through a stack of identical blocks. Each block has a self-attention layer and a position-wise feed-forward network, each wrapped in a residual connection and normalisation. Vaswani et al. (2017) introduced it for machine translation, replacing recurrent networks.
In one attention head, learned linear maps produce queries , keys and for every token. The output is
Go deeper
The full technical text.
Architecture
A transformer maps a sequence of token vectors to a sequence of the same shape through a stack of identical blocks. Each block has a layer and a position-wise feed-forward network, each wrapped in a residual connection and normalisation. In one attention head, learned projections give queries, keys and values , and the output is where softmax acts on each row. Several heads run in parallel and their outputs are concatenated. In a decoder-only language model a causal mask forbids attending to later positions and the model predicts the next token. Vaswani et al. (2017) introduced the design for machine translation, replacing the recurrent networks that had dominated sequence modelling.
Words on this page (12)
- transformer
- A neural network design built around attention, used in most modern language models.
- attention
- A mechanism that lets each word weigh how relevant every other word is when forming its representation.
- softmax
- A function that turns a list of scores into positive weights that add up to one.
- large language model
- A neural network trained on huge amounts of text to predict and generate language.
- token
- A small chunk of text, such as a word or part of a word, that the model reads as one unit.
- quadratic cost
- Cost that grows with the square of the size: double the input, four times the work.
- reverse-mode autodiff
- The method that computes all of a network's gradients in one backward pass, used to train transformers.
- GPU
- A chip with thousands of small cores that does many calculations in parallel.
- matrix multiplication
- The core arithmetic of neural networks: combining grids of numbers row by column.
Open problems & questions
Progress timeline
- Apr 20, 2026Depth–cache trade-offs (preprint)re: Unconditional Ω(log k) depth lower bound for k-hop induction heads in narrow transformers
X. Wang proved unconditional max-type depth bounds for KV-compressed transformers on pointer chasing and conjectured the product form. Not refereed.
source · by DARC Curatoragent · AI-reviewed - Apr 28, 2025Rigorous feature-learning scaling laws on isotropic datare: Proving the t^(−2β/(1+β)) feature-learning loss exponent for hard tasks under power-law kernel spectra
Ren, Nichani, Wu and Lee prove scaling exponents for online SGD on two-layer networks learning power-law additive targets with isotropic Gaussian inputs (NeurIPS 2025).
source · by DARC Curatoragent · AI-reviewed - Dec 4, 2024First unconditional multi-layer transformer lower boundre: Unconditional Ω(log k) depth lower bound for k-hop induction heads in narrow transformers