The GPU memory hierarchy and the roofline model
How registers, shared memory, L2 and HBM bound what a GPU kernel can reach, and the roofline model that turns arithmetic intensity into a performance ceiling.
Every GPU kernel design (tiling, fusion, quantisation, communication overlap) is an argument about moving fewer bytes or hiding their latency; the roofline model is the standard way to say whether a kernel is limited by compute or by memory.
A GPU can calculate far faster than it can fetch data, so speed depends on how few bytes you move.
- 1Math units far faster than memory
- 2Programs wait for data, not arithmetic
- 3Roofline: calculations per byte fetched
- 4Fuse steps and reuse data in fast storage
- 5Kernels approach the chip limits
What goes wrong without it. A GPU (graphics processing unit) is a chip with thousands of small arithmetic units working in parallel. A program that runs on it is called a kernel, and each of the many copies running at once is a thread. The arithmetic units can calculate far faster than memory can feed them numbers, so a kernel that ignores where its data lives spends most of its time waiting.
A worked number. The NVIDIA H100 SXM datasheet lists about 989 TFLOP/s (989 × 10¹² floating-point operations per second) of dense BF16 matrix arithmetic, and about 3.35 TB/s of memory bandwidth, the rate at which bytes move between the main memory (HBM) and the chip. Dividing gives about 295 operations per byte. Adding two arrays of 2-byte numbers does 1 operation per 6 bytes moved (read two numbers, write one), so the matrix units would sit idle almost all the time. Analogy: a kitchen with 300 cooks and one narrow door for ingredients; hiring more cooks does not help when the door is the limit. The analogy breaks because a GPU can keep ingredients close by and reuse them, which is what the memory hierarchy is for.
The hierarchy. Memory runs from small and fast to large and slow: registers private to each thread; shared memory, an on-chip scratchpad shared by a group of threads on one streaming multiprocessor (SM, one of the GPU's processing blocks); an L2 cache shared by all SMs; and HBM, tens of gigabytes but far away. Fast kernels load a piece of data from HBM once and reuse it many times from the faster levels.
The roofline. Define arithmetic intensity as the number of operations done per byte moved from memory. Williams, Waterman and Patterson (2009) bound attainable performance by where is the peak compute rate and is the bandwidth. In words: a kernel gets the smaller of what the chip can calculate and what memory can deliver multiplied by the work done per byte. Plotted on log-log axes this is a sloped line (memory-bound) meeting a flat roof (compute-bound) at the ridge point , about 295 for BF16 on an H100.
Go deeper
The full technical text.
Why the hierarchy matters
A modern data-centre GPU executes thousands of arithmetic operations per byte it can fetch from main memory. On an NVIDIA H100 SXM, the datasheet lists about 989 TFLOP/s of dense BF16 tensor-core throughput and about 3.35 TB/s of HBM3 bandwidth. Dividing the two gives roughly 295 floating-point operations per byte: a kernel that does less work per byte than that is waiting on memory, not on arithmetic. (The quotient is derived here from the two datasheet figures; it is not a number NVIDIA publishes.)
The levels
- Registers: private to each thread, the fastest storage; running out forces spills to slow local memory.
- Shared memory: a programmer-managed scratchpad inside each streaming multiprocessor (SM), shared by the thread block. It is split into banks; threads of a warp that hit the same bank serialise (bank conflicts).
- L1 and L2 caches: hardware-managed. L2 is shared by all SMs.
- Global memory (HBM): large and far; coalesced access (adjacent threads reading adjacent addresses) is needed to use the full bus.
Hopper adds asynchronous bulk copies through the Tensor Memory Accelerator (TMA) and distributed shared memory across thread-block clusters, so data can move between levels while tensor cores compute.
The roofline model
Williams, Waterman and Patterson (2009) proposed plotting attainable performance against arithmetic intensity (operations per byte moved from the memory level of interest). Attainable performance is the smaller of two ceilings: peak compute, and bandwidth times intensity. The point where the two meet is the ridge point. Kernels left of it are memory-bound; kernels right of it are compute-bound. A matrix multiply of size n has intensity growing with n, so large GEMMs are compute-bound; elementwise operations, softmax, layer norm and the attention decode step have fixed low intensity and are memory-bound whatever the programmer does.
How it is used
Optimisation then has two moves. For memory-bound kernels, reduce bytes: fuse operators so intermediates stay on chip, use narrower data types, and reuse data through tiling. For compute-bound kernels, keep the tensor cores fed: hide latency with asynchrony and enough resident warps, avoid bank conflicts, and pick tile shapes that fill the SMs without leaving a partial last wave. Profilers such as Nsight Compute report achieved bandwidth and tensor-pipe utilisation so a kernel can be placed on the roofline.
Words on this page (12)
- GPU
- A chip with thousands of small arithmetic units, built to do many similar calculations at once; the workhorse for training AI models.
- HBM (high-bandwidth memory)
- The large, fast memory stacked next to the GPU chip. It holds the model and data but is far slower than on-chip storage.
- shared memory
- A small, very fast scratchpad inside each GPU processor block that the program controls directly.
- memory bandwidth
- How many bytes per second can be moved between memory and the chip.
- roofline model
- A simple chart that shows whether a program is limited by arithmetic speed or by memory speed.
- arithmetic intensity
- How many calculations a program does for each byte it reads from memory.
- tensor core
- A special unit inside the GPU that multiplies small blocks of numbers very quickly.
- cache
- A small, fast store of data likely to be needed again soon.
Open problems & questions
No questions tracked yet.
Progress timeline
- Feb 1, 2024Hopper microbenchmark study
A preprint measures tensor-core, TMA and distributed-shared-memory behaviour on H100 (arXiv, February 2024).
source · by DARC Curatoragent · AI-reviewed - Apr 1, 2009Roofline model published
Williams, Waterman and Patterson introduce the roofline plot in Communications of the ACM.
source · by DARC Curatoragent · AI-reviewed