An open 4-bit pretraining recipe matching BF16 loss at 1B parameters
Difficulty 30: 2025-2026 preprints are close (0.84% at 3B on 64B tokens; 7B on 200B tokens with a BF16 tail), so the remaining work is an open, seeded run at 100B tokens rather than a new idea. Application value 60: halving GEMM time and memory traffic again on Blackwell-class hardware matters for every training run. Leverage 38: an open recipe would be reused by most open training stacks.
The problem
Can an outside group train a 1B-parameter model in 4-bit numbers with nearly 16-bit quality?
- 1Vendor reports 4-bit training works
- 2Repeat openly at 1B parameters
- 3Rotate, round randomly, keep key layers precise
- 4Compare with 16-bit over three seeds
- 5Loss within 1% of 16-bit
What is asked. Training a language model is mostly matrix multiplication. Doing those multiplications with 4-bit numbers instead of 16-bit ones lets NVIDIA's newest GPUs run them several times faster and move a quarter of the bytes. The problem asks for a fully open recipe, with public code, data and random seeds. The recipe must train a roughly 1-billion-parameter language model on 100 billion tokens of a public text corpus, with every matrix multiply in the forward and backward passes in a 4-bit format (MXFP4 or NVFP4). It must finish with a validation loss (a measure of prediction error) within 1% of the same model trained in BF16, and report results over three seeds.
Built from
earlier problems → this oneThe problems this one is built from: progress or tools it needs (builds on), special cases that come first (generalizes), and weaker results on the way (implies). On the right, what builds on it.
- Start with
- A Triton block-scaled FP4 GEMM within 10% of CUTLASS on B200Low-Precision Numerics · D31
- Then
- An open 4-bit pretraining recipe matching BF16 loss at 1B parametersThis problem
Path to understanding
foundations → problemThe research entries this problem rests on, in the order to read them. Each links to its explainer.
- Start with
- Step 2
- Step 3
- Step 4
- Microscaling (MX) and block-scaled 4-bit formatsTheory · host
- Then
- An open 4-bit pretraining recipe matching BF16 loss at 1B parametersThe problem
State of the art
NVIDIA reports a 12-billion-parameter model trained on 10 trillion tokens with NVFP4 with loss and downstream accuracy comparable to FP8 (arXiv preprint, September 2025), and the Microscaling paper reports sub-8-bit training of smaller language models. Open 4-bit work has moved quickly but none meets the bar exactly. Quartet (Castro et al., arXiv:2505.14669, MXFP4, open code) compares against FP8 rather than BF16. FP4 All the Way (Chmiel et al., arXiv:2505.19115, NVFP4) trains a 7B model on up to 200B tokens on Gaudi2 with a small loss gap that it closes by switching the backward pass to BF16 at the end. TetraJet-v2 (arXiv:2510.27527) reaches 370M parameters and 212B tokens and closes about half of the gap to BF16. Quartet II (arXiv:2601.22813) reaches 1.9B parameters on 38B tokens. Full-Stack FP4 (Ding et al., arXiv:2607.04422, preprint) reports a 0.838% loss gap to BF16 at 3B parameters on 64B tokens, with some optimizer and attention operations kept in BF16 and code availability not verified. No public result combines 100B+ tokens, a gap at most 1% against BF16, three seeds and released code.
Realistic applications
Four-bit training would roughly halve GEMM time and memory traffic again on hardware with native support; open recipes let groups without vendor teams use it.
What counts as solved
Independent reruns reproduce the loss gap within the stated bound and seed variance, from the released code.
How it gets verified
ReproductionA training run, experiment or computation rerun by someone else.
Dataset: a named public pretraining corpus (for example FineWeb) with a fixed 100B-token subset and tokenizer. Model: Llama-style decoder, about 1B parameters, config published. Metric: validation loss on a fixed held-out set at the end of training, and relative gap to the BF16 baseline run with identical data order and hyperparameters; pass if the gap is at most 1% and the three-seed spread is reported. Compute budget: on the order of 1 to 2 thousand GPU-hours on 4-bit capable hardware or emulation (emulation is allowed but must be labelled). Seeds: 3. Artefacts: code, configs, loss logs and checkpoints at a pinned commit.
Claims under verification
+ add a claimNo claimed results on the Verification board for this problem yet.
Where the work is
Dominated by compute for training runs, with recipe design and kernel work.
Bottlenecks
ranked by votes- #1ComputeCost of 100B-token training runs with seeds
Three seeds each for BF16 and the 4-bit recipe is thousands of GPU-hours.
Community grades
Sign in to add your own grade.
Connected problems
- This problem builds on A Triton block-scaled FP4 GEMM within 10% of CUTLASS on B200Fast training depends on an efficient open block-scaled GEMM kernel.
Progress
- Jul 5, 20260.84% gap at 3B parameterssource
Full-Stack FP4 reports final loss 2.286 versus 2.267 for BF16 on a 3B model trained on 64B tokens (preprint).
- May 25, 20257B model trained mostly in FP4source
FP4 All the Way trains a 7B model with NVFP4 on up to 200B tokens, closing a small loss gap with a BF16 backward phase at the end (preprint).