A bitwise-reproducible all-reduce independent of topology with under 5% overhead
Difficulty 39: fixed-order tree reductions already give bitwise determinism (TP-invariant inference, OPEN-1B), but doing it at bandwidth-optimal speed across 1 MB to 1 GB pulls numerics and systems design together. Application value 37: exact restarts, debugging, auditable training and removing train-inference mismatch in RL, not raw speed. Leverage 31: reproducible and auditable training builds on it.
The problem
Can GPUs add up numbers in a way that always gives exactly the same bits, without slowing down?
- 1Rounding makes grouping change results
- 2Different patterns give different bits
- 3Large runs cannot be restarted exactly
- 4Fix the order or add exactly
- 5Same bits, within 5% of NCCL speed
What is asked. When many GPUs train one model, they repeatedly add up their results with an all-reduce: every GPU contributes a tensor and receives the sum. Floating-point addition is not associative. can differ from in the last bits, because each addition rounds. So the sum's exact bits depend on the order in which partial sums are combined, and that order changes with the algorithm, the number of GPUs and how they are wired. The problem asks for an all-reduce (and reduce-scatter) whose output is across repeated runs, across ring versus tree algorithms, and across different assignments of ranks to GPUs. It must handle BF16 and FP32 sums, and run within 5% of NCCL's speed for messages from 1 MB to 1 GB on 8 H100s.
Path to understanding
foundations → problemThe research entries this problem rests on, in the order to read them. Each links to its explainer.
- Start with
- Step 2
- PyTorchLibrary
- Step 3
- Then
- A bitwise-reproducible all-reduce independent of topology with under 5% overheadThe problem
State of the art
Floating-point addition is not associative, so the result of a reduction depends on the order in which partial sums are combined. For a fixed NCCL configuration, a given algorithm and rank order generally produce a fixed order, but different algorithms, protocols, rank counts or topologies give different bits, and library documentation notes that results are not guaranteed to match across configurations. NCCL offers no topology-independent deterministic mode. Order-independent summation schemes exist (higher-precision or exactly-rounded accumulation). Two recent systems fix the reduction order instead: Zhang et al. (arXiv:2511.17826) make LLM inference bitwise identical across tensor-parallel sizes with a fixed binary-tree all-reduce written in Triton, and OPEN-1B (arXiv:2609.17380) makes a training run bitwise reproducible across heterogeneous hardware with a fixed-order local sum after an all-to-all. Neither reports bus bandwidth against NCCL across 1 MB to 1 GB, so the cost of determinism at training-scale message sizes is not established.
Realistic applications
Debugging, regression testing and exact restart of large training runs are easier when parallel configuration changes do not change the numerics.
What counts as solved
Public code and a test suite showing bit equality across the listed configurations and overhead under the bound, reproduced independently.
How it gets verified
GPU benchmarkA kernel run and timed on stated hardware against a baseline.
Hardware: one node with 8x H100 SXM 80 GB. Operation: sum all-reduce and reduce-scatter, BF16 and FP32, sizes 1 MB to 1 GB (powers of 4), random normal inputs and adversarial inputs with mixed magnitudes. Reproducibility test: outputs bitwise equal across 100 repeated runs, across NCCL_ALGO in {Ring, Tree} for the baseline (to show it differs), and across at least 5 random permutations of rank-to-GPU assignment, for the candidate. Baseline: NCCL latest stable, pinned, default settings, measured with nccl-tests. Metric: bus bandwidth, median of 100 iterations; pass if at least 95% of NCCL's at every size. Harness: public script and test suite at a pinned commit.
Claims under verification
+ add a claimNo claimed results on the Verification board for this problem yet.
Where the work is
Numerics research on summation order combined with systems engineering.
Bottlenecks
ranked by votes- #1TheoryOrder-independent summation at low cost
Exact or fixed-order accumulation needs wider accumulators or extra passes; their overhead against a ring at large sizes is unquantified.
Community grades
Sign in to add your own grade.
Connected problems
No connections yet.
Progress
- Nov 21, 2025Tree all-reduce for TP-invariant inferencesource
A fixed binary-tree all-reduce makes outputs bitwise identical across tensor-parallel sizes (preprint).