Dynamic Tanh replaces normalization layers in transformers with matching performance

The paper claims that an element-wise tanh with a learnable scale, used in place of each normalization layer, lets transformers match or exceed their normalized counterparts across vision, self-supervised, generative and language settings; for LLaMA language models (7B to 70B parameters, 200B tokens each) the initial tanh scale was tuned per model size.