Low-Precision Numerics

FP8, FP4 and integer formats, and the scaling and rounding that keep low-precision training and inference accurate.

Open problems, most leverage first

All open problems →

#VotesProblemWork mixDVLStatus

010An open 4-bit pretraining recipe matching BF16 loss at 1B parametersReproductionMicroscaling (MX) and block-scaled 4-bit formats1 link3 bottlenecks
306038OpenD 30V 60L 38▲ 0
020FP8 GEMM with FP32-accumulation accuracy at long K without a throughput penaltyGPU benchmarkFP8 formats and scaling recipes for training and inference2 links3 bottlenecks
344726OpenD 34V 47L 26▲ 0
030A Triton block-scaled FP4 GEMM within 10% of CUTLASS on B200GPU benchmarkMicroscaling (MX) and block-scaled 4-bit formats1 link2 bottlenecks
314323OpenD 31V 43L 23▲ 0

Research

Filter in Research →

Low-Precision Numerics</> 3 reposTheoryEmerging

Microscaling (MX) and block-scaled 4-bit formats

Data formats that share one scale across a small block of narrow elements (MXFP8, MXFP6, MXFP4, MXINT8, and NVIDIA's NVFP4), implemented in Blackwell tensor cores.

2 problems0 dependents0 maintainers

Low-Precision Numerics</> 2 reposTheoryActive

FP8 formats and scaling recipes for training and inference

The two 8-bit floating-point encodings (E4M3 and E5M2), why tensors need scale factors, and the recipes (per-tensor, delayed, fine-grained blockwise) that make FP8 GEMMs usable for LLMs.