■ Block 01 / research
Research
Foundations other work depends on. Each entry explains what it is, why it is load-bearing, its open problems, and a reviewed timeline of progress.
Foundations other work depends on. Each entry explains what it is, why it is load-bearing, its open problems, and a reviewed timeline of progress.
How large models are split across GPUs (tensor, pipeline, data/ZeRO, sequence and context parallelism) and how fused kernels hide communication behind matrix multiplication.
The collective operations (all-reduce, all-gather, reduce-scatter) that synchronise GPUs in training, the ring and tree algorithms behind them, and NVIDIA's NCCL library that implements them.