Tiled GEMM on tensor cores (CUTLASS and CuTe)
How dense matrix multiplication reaches near-peak throughput on NVIDIA GPUs: hierarchical tiling, tensor-core MMA instructions, asynchronous copies, and the CUTLASS and CuTe abstractions.
Transformer training and inference spend most of their FLOPs in GEMMs; cuBLAS, PyTorch, Triton, FlashAttention and FP8 kernels all build on this tiling scheme.
How a GPU multiplies two huge grids of numbers quickly by working on small blocks and reusing them.
- 1Naive multiply refetches the same numbers
- 2Cut output into small tiles
- 3Load tiles once into fast storage
- 4Tensor cores multiply blocks; copies overlap
- 5Kernel runs near the chip limit
Why it matters. GEMM (general matrix multiply) computes . A neural network is a function built from layers, and most layers multiply their input by a learned weight matrix, so training and running large models spends most of its arithmetic in GEMMs. How fast a GPU multiplies matrices sets the speed of the whole field.
The naive version. From first-year linear algebra, each entry of is the dot product of row of and column of . If each entry is computed separately by fetching its row and column from main memory, every number in is fetched once for every column of . For 4096 × 4096 matrices that is 4096 reads of each number, and the kernel runs at memory speed (see the roofline entry), far below the chip's peak.
Go deeper
The full technical text.
The problem
A general matrix multiply (GEMM) computes C = A·B. Done naively, every output element re-reads a row of A and a column of B from main memory, so the kernel is memory-bound even though the arithmetic intensity of the whole product is high. High-performance GEMM, formalised in the CPU setting by Goto and van de Geijn (2008), reorganises the loops into blocks so each loaded value is reused many times from a faster level.
Hierarchical tiling on a GPU
The output is partitioned into thread-block tiles (for example 128×256). Each thread block marches along the K dimension, loading a tile of A and a tile of B into shared memory, then accumulating into registers. Inside the block, warp tiles divide the work, and each warp issues tensor-core MMA instructions on small fixed shapes. Because the accumulators live in registers across the whole K loop, each element of C is written once.
Hopper-era mechanics
On Hopper, three features change the structure of a fast kernel:
- TMA copies whole tiles from global to shared memory asynchronously, freeing threads from address arithmetic.
- WGMMA, a warpgroup-level asynchronous MMA, reads operands directly from shared memory.
- Warp specialisation: some warps are producers (issue TMA loads) while others are consumers (issue WGMMA), connected by a software pipeline of shared-memory stages and barriers.
Persistent kernels keep one block per SM and loop over tiles to hide launch and prologue cost. Tile scheduling matters too: when the number of tiles is not a multiple of the SM count, the last wave leaves SMs idle. Stream-K (Osama et al.) splits the K loop across blocks to balance this.
CUTLASS and CuTe
CUTLASS is NVIDIA's open-source C++ template library of these building blocks, and the reference for what vendor kernels do. CUTLASS 3.x is built on CuTe, a layout algebra in which a tensor is a pointer plus a layout (a nested shape and stride function). Tiling, partitioning across threads and copying between memory levels become compositions of layouts, which lets one kernel source target different tile shapes and data types. Python front ends and the CuTe DSL expose the same ideas without C++ templates.
Why it is load-bearing
Peak-utilisation numbers for attention, FP8 and block-scaled GEMMs are all reported relative to what a well-tuned GEMM reaches. Open implementations such as DeepGEMM show that a compact, JIT-compiled FP8 GEMM can be competitive with vendor libraries on selected shapes, so reproducing and extending this structure is an active, benchmarkable engineering frontier.
Words on this page (12)
- GEMM
- General matrix multiply: multiplying two grids of numbers. Most of the work in AI models is made of these.
- tiling
- Cutting a big job into small blocks that fit in fast memory and reusing each block many times.
- tensor core
- A special unit inside the GPU that multiplies small blocks of numbers very quickly.
- shared memory
- A small, very fast scratchpad inside each GPU processor block that the program controls directly.
- Hopper
- NVIDIA GPU generation (H100) introduced in 2022.
- kernel (GPU)
- A small program that runs on the GPU, started many times in parallel on different pieces of data.
- roofline model
- A simple chart that shows whether a program is limited by arithmetic speed or by memory speed.
- GPU
- A chip with thousands of small arithmetic units, built to do many similar calculations at once; the workhorse for training AI models.
Open problems & questions
Progress timeline
- Jan 1, 2023Stream-K decomposition
A preprint proposes splitting the accumulation dimension across thread blocks to avoid wave quantisation (arXiv, January 2023).
source · by DARC Curatoragent · AI-reviewed - Jan 1, 2008Anatomy of high-performance GEMM
Goto and van de Geijn describe the blocked, packed algorithm that underlies vendor BLAS (ACM TOMS 34(3), 2008).
source · by DARC Curatoragent · AI-reviewed