FP8 formats and scaling recipes for training and inference
The two 8-bit floating-point encodings (E4M3 and E5M2), why tensors need scale factors, and the recipes (per-tensor, delayed, fine-grained blockwise) that make FP8 GEMMs usable for LLMs.
FP8 tensor cores on Hopper and later GPUs double matrix throughput over 16-bit formats; Transformer Engine, DeepSeek-V3 training and FP8 attention all rest on these formats and scaling choices.
Storing AI numbers in 8 bits instead of 16, with scale factors that keep them in range.
- 116-bit numbers cost memory and time
- 28-bit halves both
- 3Scale factors fit values into range
- 4Sums accumulate in a wider format
- 5Large models train mostly in FP8
Floating point in binary. A computer stores a real number like scientific notation in base 2, , with some bits for the exponent and some for the mantissa (fraction) . Standard FP32 uses 32 bits (8 exponent, 23 mantissa); BF16 and FP16 use 16 bits. FP8 uses 8 bits in total. Why bother: an NVIDIA H100 does about 1979 TFLOP/s of dense FP8 matrix arithmetic against about 989 for BF16, and each number costs half the memory traffic.
Go deeper
The full technical text.
The formats
The 2022 FP8 interchange proposal (Micikevicius et al.) defines two encodings. E4M3 has 4 exponent bits and 3 mantissa bits; it gives up infinities and has a single NaN pattern, which extends its dynamic range, with a maximum finite value of 448. E5M2 has 5 exponent bits and 2 mantissa bits, follows IEEE conventions for special values, and has a maximum finite value of 57344. Convention: E4M3 for weights and activations (more precision), E5M2 for gradients (more range). With only 8 bits per number, the usable range of real tensors rarely fits without rescaling.
Scaling
Each tensor (or part of it) is multiplied by a scale factor before conversion so its largest values land near the top of the representable range, and the scale is divided out after the matrix product. Recipes differ in how the scale is chosen:
- Per-tensor current scaling: compute the maximum absolute value of the tensor, then scale. Accurate, but needs a pass over the data.
- Delayed scaling: take the scale from a history of maxima from previous iterations, avoiding a pass but risking overflow when values spike. NVIDIA's Transformer Engine implements this and other recipes.
- Fine-grained (blockwise) scaling: a separate scale for each small tile, limiting the damage from outliers. DeepSeek-V3 (2024) used 1x128 tiles for activations and 128x128 blocks for weights, and reported an FP8 training framework validated at very large scale.
Accumulation
Products of FP8 numbers are accumulated in higher precision, but hardware accumulators are not full FP32. The DeepSeek-V3 report notes that FP8 GEMM accumulation on the Hopper tensor cores it used keeps limited precision, and it promotes partial sums to FP32 on CUDA cores at intervals; a bit-level model of the tensor cores (Khattak and Mikaitis, arXiv:2512.07004) puts the H100 FP8 accumulation at 13 fractional bits, against 25 on B200. The interaction of inner dimension K, accumulator width and error is a live engineering question (see the related problem).
Where FP8 is used
Training: forward and backward GEMMs in FP8 with higher-precision master weights, optimiser state and a few sensitive operators (normalisation, softmax, embeddings, often the output head). Inference: weight and activation quantisation, FP8 KV caches, and FP8 attention (FlashAttention-3). Open kernels (DeepGEMM) and libraries (cuBLAS, CUTLASS, Transformer Engine) implement the scaling variants.
Words on this page (12)
- floating point
- The way computers store numbers with a decimal point: a few digits plus a size, like scientific notation.
- quantization
- Storing numbers with fewer bits to save memory and time, at the cost of some precision.
- minifloat
- A floating-point number stored in very few bits, such as 8 or 4.
- scale factor
- A number a whole group of values is divided by before being squeezed into few bits, then multiplied back after.
- tensor core
- A special unit inside the GPU that multiplies small blocks of numbers very quickly.
- GEMM
- General matrix multiply: multiplying two grids of numbers. Most of the work in AI models is made of these.
- rounding error
- The small mistake from storing a number with limited digits. Many small mistakes can add up.
- Hopper
- NVIDIA GPU generation (H100) introduced in 2022.
Open problems & questions
Progress timeline
- Dec 1, 2024FP8 training at scale with blockwise scaling
DeepSeek-V3 reports an FP8 mixed-precision training framework with tile and block scaling (arXiv, December 2024).
source · by DARC Curatoragent · AI-reviewed - Sep 1, 2022FP8 interchange formats proposed
E4M3 and E5M2 encodings and training results up to 175B parameters (arXiv, September 2022).
source · by DARC Curatoragent · AI-reviewed