A fused GEMM + reduce-scatter kernel hiding 90% of communication on 8 H100 GPUs
Difficulty 36: fused overlap kernels exist (FLUX reports up to 96% overlap in its own setting), but a uniform 0.9 efficiency including latency-bound small M is demanding. Application value 52: exposed tensor-parallel communication costs a double-digit share of layer time in some configurations. Leverage 21: the technique transfers to other collective-plus-GEMM pairs but gates little.
The problem
Can the sum between GPUs be hidden almost entirely behind the multiplication that produces it?
- 1Partial answers must be summed
- 2Sequential: multiply, then wait
- 3Split into tiles; send each when done
- 4Fused kernel overlaps both jobs
- 590% of waiting time hidden
What is asked. Large models are split across several GPUs. In tensor parallelism, each of the 8 GPUs in a server holds a slice of a weight matrix and computes a partial product. The partial results must then be summed across GPUs, and each GPU keeps one slice of the total (a ). Done the simple way, each GPU multiplies, then waits while the data is exchanged. The problem asks for one fused kernel on 8 H100s that starts sending pieces of the result while the rest is still being computed. It must hide at least 90% of the communication time on every shape in a fixed list, and its results must match the unfused path.
Path to understanding
foundations → problemThe research entries this problem rests on, in the order to read them. Each links to its explainer.
- Start with
- Step 2
- PyTorchLibrary
- The transformer architectureAlgorithm
- Tiled GEMM on tensor cores (CUTLASS and CuTe)Algorithm
- Step 3
- Step 4
- Then
- A fused GEMM + reduce-scatter kernel hiding 90% of communication on 8 H100 GPUsThe problem
State of the art
The FLUX preprint states that up to 96% of communication can be overlapped by a fused kernel and reports end-to-end speedups of up to 1.24x for training over Megatron-LM on a 128-GPU cluster; Triton-distributed (preprint) reports effective overlap on up to 64 devices. These figures are for the authors' configurations. A uniform 0.9 efficiency across a sweep of shapes (including small M, where communication is latency-bound) with a stated definition has not been shown in a primary source known to the curator.
Realistic applications
Exposed tensor-parallel communication costs a double-digit share of layer time in some configurations; hiding it raises utilisation without changing the model.
What counts as solved
Public kernel and harness at a pinned commit meeting the 0.9 threshold on all shapes, reproduced on a second node.
How it gets verified
GPU benchmarkA kernel run and timed on stated hardware against a baseline.
Hardware: 8x H100 SXM 80 GB in one NVLink/NVSwitch node. Operation: row-parallel linear (GEMM then reduce-scatter) in BF16, FP32 accumulation, TP degree 8. Shapes: tokens M in {512, 2048, 8192, 32768}, (N, K) from the Llama-3 70B and Mixtral 8x22B projections. Baselines: cuBLAS (CUDA 12.8 or later) GEMM followed by NCCL (latest stable, pinned) reduce-scatter on the same stream, and each alone for T_comm and T_gemm. Metric: overlap efficiency as defined, median of 200 iterations after warmup. Correctness: output within BF16 tolerance (max relative error at most 1.1x the unfused path versus an FP32 reference). Harness: public script at a pinned commit.
Claims under verification
+ add a claimNo claimed results on the Verification board for this problem yet.
Where the work is
Systems engineering of tile scheduling and synchronisation; some measurement.
Bottlenecks
ranked by votes- #1EngineeringFine-grained synchronisation between compute and communication tiles
Tile completion signals and remote writes need low-overhead flags without slowing the GEMM.
Community grades
Sign in to add your own grade.
Connected problems
No connections yet.