Distributed Training & Communication

Collectives, parallelism strategies and interconnects that spread one model across thousands of accelerators.

Open problems, most leverage first

All open problems →

#VotesProblemWork mixDVLStatus

010A bitwise-reproducible all-reduce independent of topology with under 5% overheadGPU benchmarkNCCL and collective communication: ring and tree all-reduce2 bottlenecks
393731OpenD 39V 37L 31▲ 0
020A fused GEMM + reduce-scatter kernel hiding 90% of communication on 8 H100 GPUsGPU benchmarkTensor, data and pipeline parallelism with compute-communication overlap3 bottlenecks
365221OpenD 36V 52L 21▲ 0

Research

Filter in Research →

Distributed Training & Communication</> 2 reposAlgorithmActive

Tensor, data and pipeline parallelism with compute-communication overlap

How large models are split across GPUs (tensor, pipeline, data/ZeRO, sequence and context parallelism) and how fused kernels hide communication behind matrix multiplication.

1 problem0 dependents0 maintainers

Distributed Training & Communication</> 2 reposLibraryMature

NCCL and collective communication: ring and tree all-reduce

The collective operations (all-reduce, all-gather, reduce-scatter) that synchronise GPUs in training, the ring and tree algorithms behind them, and NVIDIA's NCCL library that implements them.