Normalization-free language models at 7B parameters and beyond
Difficulty 25: a bounded paired experiment; the DyT paper already reached 70B parameters at 200B tokens, so the open part is the 1T-token horizon, which is cost rather than ideas. Application value 22: removes one memory-bound reduction per sub-layer. Leverage 32: a clean answer says which of normalization's proposed roles matters, which feeds parametrization and optimizer work.
The problem
Does a language model without normalization layers still train as well when run for a trillion tokens or more?
- 1DyT swaps norm layers for tanh
- 2Matched RMSNorm at 7B–70B, 200B tokens
- 3Trillion-token runs never tested
- 4Paired 7B, 1T-token runs decide
What is being asked. Almost every large language model contains normalization layers: after each block, the vector of activations is divided by its root-mean-square size so its entries stay of order one. Without them, numbers can grow layer after layer until training blows up, much like a feedback amplifier with gain above one. Dynamic Tanh (DyT) replaces each normalization layer with a simple element-wise function, . Here squashes every entry into the range −1 to 1, is a learned "stretch" factor, and , are learned per-channel scale and shift. No statistics over the vector are computed, which saves a small reduction step in every layer. The question is whether a 7-billion-parameter model with DyT, trained on at least 1 trillion tokens (text chunks), ends with a validation loss (prediction error on held-out text) within 0.5% of an identical model that uses RMSNorm, with no sudden loss spikes.
Path to understanding
foundations → problemThe research entries this problem rests on, in the order to read them. Each links to its explainer.
- Start with
- Reverse-mode automatic differentiationAlgorithm
- Fast matrix multiplicationAlgorithm
- Step 2
- The transformer architectureAlgorithm
- Step 3
- Normalization layers (BatchNorm, LayerNorm, RMSNorm)Algorithm · host
- Then
- Normalization-free language models at 7B parameters and beyondThe problem
State of the art
The DyT paper pre-trains LLaMA 7B, 13B, 34B and 70B models on 200B tokens each and reports losses and zero-shot averages matching RMSNorm (for 7B, final loss 1.59 with RMSNorm versus 1.60 with DyT). For language models the initial scale α₀ had to be tuned per size and separately for attention and other layers (from 0.8/0.2 at 7B down to 0.2/0.05 at 34B and 70B), using 30B-token search runs; the paper's claim of working without retuning applies mainly to its vision and diffusion settings. Follow-ups are small-scale: Derf (Chen et al., arXiv:2512.10938) reports DyT slightly behind LayerNorm on GPT-2 124M (loss 2.97 versus 2.94), and the BHyT preprint (arXiv:2601.09719) argues DyT is fragile at depth. No normalization-free language model trained on 1T tokens or more, and no independent reproduction of the 7B result, was found. Normalization placement is known to change trainability (Xiong et al., 2020), so long-horizon stability is the open part.
Realistic applications
Removing normalization removes a memory-bound operation per sub-layer and a source of reduction-synchronization in tensor-parallel training; the size of the saving has not been measured in the sources used here. The larger value is scientific: it would show which of normalization's several proposed roles is actually needed.
What counts as solved
A public report with code and logs of paired runs (DyT versus tuned RMSNorm, same data order, at least 3 seeds each at a smaller proxy scale and one run each at 7B) showing the validation-loss and benchmark parity above, or a documented failure (divergence or a gap above the threshold) that survives a tuning sweep of the DyT initial scale.
How it gets verified
ReproductionA training run, experiment or computation rerun by someone else.
Dataset: a fixed public pre-training corpus (for example FineWeb or DCLM) with identical token order for both arms. Metric: held-out validation loss and the average of a fixed benchmark suite (for example the lm-evaluation-harness tasks used by the baseline's own paper). Threshold: DyT within 0.5% relative validation loss of tuned RMSNorm and no divergence. Budget: at least 3 seeds at a proxy scale (about 400M parameters) and one paired run at 7B parameters and 1T tokens or more, which is roughly the 6·N·D rule of 4×10^22 FLOPs per arm. Report the DyT initial-scale sweep.
Claims under verification
+ add a claimWhere the work is
A large paired training comparison; theory plays a small part in choosing the initial scale.
Bottlenecks
ranked by votes- #1ComputeCost of a paired 7B, 1T-token run
Two runs at 7B and 1T tokens or more are of the order of 10^22 FLOPs each, beyond most academic budgets.
Community grades
Sign in to add your own grade.
Connected problems
No connections yet.
Progress
- Mar 13, 2025DyT proposedsource
Zhu et al. report parity with normalized transformers across settings.