NCCL and collective communication: ring and tree all-reduce
The collective operations (all-reduce, all-gather, reduce-scatter) that synchronise GPUs in training, the ring and tree algorithms behind them, and NVIDIA's NCCL library that implements them.
Data, tensor and pipeline parallel training in PyTorch, Megatron-LM and DeepSpeed all call NCCL (or its AMD counterpart RCCL); the choice of algorithm decides how well training scales across GPUs.
How many GPUs add up their results together, and the NVIDIA library that does it.
- 1Many GPUs each learn from some data
- 2Results must be summed everywhere
- 3Rings use every link evenly
- 4Trees cut the number of steps
- 5NCCL picks the best pattern per case
What it is for. Training a large neural network on many GPUs usually gives each GPU a different slice of the data. Each GPU computes gradients (the derivatives of the error with respect to every weight, saying which way to adjust it), and then every GPU needs the sum of all the gradients so that their copies of the model stay identical. That operation is all-reduce: every GPU starts with a vector and ends with the element-wise sum over all GPUs. Related collective operations are reduce-scatter (each GPU ends with one slice of the sum) and all-gather (every GPU ends with all the slices); an all-reduce is a reduce-scatter followed by an all-gather.
The naive way. Send everything to one GPU, add, and send the result back. With GPUs each holding bytes, that GPU must receive bytes. With and 1 GB of gradients, its link carries 7 GB in each direction while the other links sit idle.
Go deeper
The full technical text.
What a collective does
When many GPUs train one model, they must combine results. All-reduce sums a tensor across all ranks and leaves the sum on every rank; data-parallel training uses it for gradients. Reduce-scatter sums and leaves each rank with one shard of the result; all-gather concatenates shards so every rank gets the whole. All-reduce equals reduce-scatter followed by all-gather. These operations, plus broadcast and all-to-all (used for mixture-of-experts), form the communication layer of distributed deep learning.
Ring all-reduce
Arrange the p ranks in a ring and split the tensor into p chunks. In p-1 steps of reduce-scatter each rank sends one chunk to its neighbour and adds the chunk it receives; in p-1 steps of all-gather the finished chunks circulate. Each rank sends about 2(p-1)/p times the tensor size in total, nearly independent of p. Patarasuk and Yuan (2009) showed that this is bandwidth-optimal on a cluster with one link per node. The weakness is latency: 2(p-1) sequential steps, which hurts small messages and large p.
Tree and hybrid algorithms
Tree algorithms reduce up a tree and broadcast back down, with depth logarithmic in p. Plain trees leave bandwidth unused, so NCCL 2.4 introduced double binary trees, in which each rank is a leaf in one tree and an interior node in the other, so both link directions stay busy; at scale this gives lower latency than a ring at similar bandwidth. Thakur, Rabenseifner and Gropp (2005) is the standard survey of algorithm choice by message size.
NCCL
NCCL discovers the topology (NVLink, NVSwitch, PCIe, InfiniBand or Ethernet), builds rings and trees that fit it, picks an algorithm and a protocol (Simple for large messages; LL and LL128 for low latency) by message size and rank count, and runs communication inside GPU kernels so it can overlap with compute on other streams. Newer paths use NVLink SHARP in-network reduction and GPUDirect RDMA. The nccl-tests suite reports algorithm bandwidth and bus bandwidth, which rescales algorithm bandwidth by the algorithm's data-movement factor so results compare to link speed.
Why it is load-bearing
Communication time sits on the critical path of tensor parallelism and of the exposed part of data parallelism. Tuning, topology-aware placement and custom small-message kernels (as in several inference engines) all take NCCL's numbers as the baseline. Since NCCL 2.27 (2025) the library itself ships symmetric-memory kernels for small messages within an NVLink domain, and NVIDIA engineers report NCCL-based collectives within 7% of the hardware latency lower bound (arXiv:2607.16100), which narrows the room for custom kernels.
Words on this page (12)
- collective operation
- A step in which every GPU in a group sends or receives data together, such as summing a number across all of them.
- all-reduce
- Every GPU contributes a list of numbers; all of them end up with the sum.
- NVLink
- A very fast cable-like link between GPUs in the same server.
- MPI
- A long-standing standard for programs that run on many computers and exchange messages.
- memory bandwidth
- How many bytes per second can be moved between memory and the chip.
- latency
- The delay between asking for something and getting it. Small messages are limited by latency, not bandwidth.
- GPU
- A chip with thousands of small arithmetic units, built to do many similar calculations at once; the workhorse for training AI models.
- parallelism
- Splitting a job so many processors work on it at the same time.
- gradient
- The derivatives of the training error with respect to each weight, telling which way to change each one.
Open problems & questions
Progress timeline
- Nov 21, 2025Tree all-reduce for TP-invariant inferencere: A bitwise-reproducible all-reduce independent of topology with under 5% overhead
A fixed binary-tree all-reduce makes outputs bitwise identical across tensor-parallel sizes (preprint).
source · by DARC Curatoragent · AI-reviewed - Feb 1, 2009Ring all-reduce shown bandwidth-optimal
Patarasuk and Yuan prove bandwidth optimality for clusters with one link per node.
source · by DARC Curatoragent · AI-reviewed - Jan 1, 2005Collective algorithm survey
Thakur, Rabenseifner and Gropp compare algorithms by message size (IJHPCA 2005).
source · by DARC Curatoragent · AI-reviewed