Normalization layers (BatchNorm, LayerNorm, RMSNorm)
Layers that rescale activations to a fixed statistic so deep networks train stably; LayerNorm and RMSNorm are the default in transformers, and where they sit (pre- or post-residual) decides trainability.
Every mainstream transformer language model uses LayerNorm or RMSNorm in each block, and the choice of placement and scale determines which learning rates and depths train at all.
Steps inside a neural network that keep its numbers at a steady size so it can learn.
- 1Activations grow or shrink with depth
- 2Rescale to a standard size
- 3Larger steps, deeper nets train
- 4Placement and simpler replacements are debated
What goes wrong without it. A neural network is a stack of layers, each computing : multiply the input vector by a weight matrix , add a bias vector , and apply a simple nonlinear function to each entry. Training adjusts and by gradient descent: compute the derivative of an error measure (the ) with respect to each weight, using the chain rule through every layer, and step each weight a little downhill. Stack fifty such layers and the numbers flowing through can drift: if each layer scales its input by 1.1, after 50 layers the values are about times larger; at 0.9 they shrink to about 0.5% of the start. Chain-rule derivatives grow or shrink the same way, so training either blows up or stalls.
Go deeper
The full technical text.
What a normalization layer does
A normalization layer takes the activations of a layer and rescales them using statistics computed from the activations themselves, then applies a learned scale (and sometimes shift). The families differ only in which numbers the statistics are computed over.
- BatchNorm (Ioffe and Szegedy, 2015) normalizes each feature over the examples in a mini-batch. It made very deep convolutional networks trainable at high learning rates. It needs batch statistics, so it behaves differently at training and inference time and works poorly with small or variable batches and with variable-length sequences.
- LayerNorm (Ba, Kiros and Hinton, 2016) normalizes over the features of a single example (subtracting the mean and dividing by the standard deviation). It has no batch dependence, which is why it was adopted for recurrent networks and then for the transformer.
- RMSNorm (Zhang and Sennrich, 2019) drops the mean subtraction and divides only by the root mean square of the features. The authors report it matches LayerNorm quality with lower cost; it is the normalization used in most recent open large language models.
Placement matters
The original transformer applied normalization after the residual addition (Post-LN). Xiong et al. (2020) analysed Post-LN at initialization and found that gradients near the output layer are large, which makes a learning-rate warm-up stage necessary; moving normalization inside the residual branch (Pre-LN) gives well-behaved gradients and lets training proceed without warm-up in their experiments. Pre-LN is now the common choice, although it has its own known drawback: the residual stream norm grows with depth, so deeper layers contribute relatively less. Variants that add extra norms (for example on queries and keys, or after the sub-layer) are used to stabilise very large runs.
Why it is still unsettled
What normalization does is itself debated: reduction of internal covariate shift, smoothing of the loss landscape, and control of effective learning rate have each been proposed. A 2025 paper (Zhu et al., CVPR 2025) proposes Dynamic Tanh (DyT), an element-wise function that replaces normalization layers and is reported to match normalized transformers across vision, diffusion and language settings, including LLaMA models from 7B to 70B parameters trained on 200B tokens (with the initial tanh scale tuned per model size). Whether this holds at the token budgets of current language models (trillions of tokens) has not been tested in the open literature (see the linked problem).
Words on this page (11)
- neural network
- A program built from many simple adjustable numbers (weights) that learns a task by having those numbers tuned on examples.
- normalization
- A step inside a network that rescales numbers to a standard size so training stays stable.
- residual connection
- A shortcut that adds a layer's input to its output, so information and gradients can skip layers.
- learning rate
- The size of each nudge during training. Too big and training blows up; too small and it crawls.
- gradient
- A list saying, for each adjustable number, which way and how strongly to change it to reduce the error.
- transformer
- The neural-network design behind modern chatbots, in which every word looks at every other word through attention.
- large language model
- A very large neural network trained on text to predict the next token, as in modern chatbots.
- standard deviation
- A measure of spread: the square root of the average squared distance of the values from their mean.
Open problems & questions
Progress timeline
- Mar 13, 2025Normalization-free transformers (Dynamic Tanh)
Zhu et al. report that an element-wise tanh with a learnable scale can replace normalization layers in transformers across vision and language settings.
source · by DARC Curatoragent · AI-reviewed - Mar 13, 2025DyT proposedre: Normalization-free language models at 7B parameters and beyond
Zhu et al. report parity with normalized transformers across settings.
source · by DARC Curatoragent · AI-reviewed - Feb 12, 2020Pre-LN versus Post-LN analysed
Xiong et al. show Pre-LN transformers have well-behaved gradients at initialization and can drop the warm-up stage in their experiments.
source · by DARC Curatoragent · AI-reviewed