A Triton BF16 GEMM within 3% of cuBLAS across an LLM shape sweep on H100
Difficulty 27: compiler heuristics and scheduling on a well-understood operation; slightly above the FP8 parity task (24) because the kernel author cannot hand-place warps and the tuning budget is capped. Application value 45: matters wherever fused epilogues are written in Python. Leverage 30: decides how close torch.compile output gets to vendor GEMMs.
The problem
Can a Python-level Triton matrix multiply match the closed vendor library within 3% on 40 real sizes?
- 1cuBLAS is closed and heavily tuned
- 2Real shapes are thin, wide and awkward
- 3Triton needs Hopper features and tuning
- 4At most 20 settings per shape
- 5Python-level code at vendor speed
What is asked. Triton lets a programmer write a GPU kernel in Python for whole tiles of a matrix, and a compiler handles the details of threads and memory (see the Triton entry). PyTorch's compiler generates Triton code automatically. The problem asks whether a Triton matrix multiply in BF16 (16-bit floats, with sums kept in 32-bit) can run at 97% or more of NVIDIA's cuBLAS library on one H100. It must do this on every one of 40 shapes taken from real language models. Autotuning may try at most 20 configurations per shape.
Built from
earlier problems → this oneThe problems this one is built from: progress or tools it needs (builds on), special cases that come first (generalizes), and weaker results on the way (implies). On the right, what builds on it.
- Start with
- A Triton BF16 GEMM within 3% of cuBLAS across an LLM shape sweep on H100This problem
- Then
- An autotuner that finds configurations within 2% of the best in 20 trialsKernel Languages & Compilers · D41
Path to understanding
foundations → problemThe research entries this problem rests on, in the order to read them. Each links to its explainer.
- Start with
- Step 2
- Tiled GEMM on tensor cores (CUTLASS and CuTe)Algorithm
- PyTorchLibrary
- Step 3
- Then
- A Triton BF16 GEMM within 3% of cuBLAS across an LLM shape sweep on H100The problem
State of the art
Triton's tutorial GEMM and tuned variants reach a large fraction of cuBLAS on large square shapes in public benchmarks, and recent Triton releases support TMA and warp-specialised pipelines on Hopper. The gap on skinny, odd-sized and persistent-scheduling cases is reported by kernel authors but the curator found no primary source that fixes a single number across a full sweep. The 3% bar with a bounded tuning budget is a deliberate stretch; it should be confirmed as open against current Triton benchmarks.
Realistic applications
If a compiler-language GEMM reaches vendor parity, fused GEMM epilogues and custom quantisation schemes can be written without leaving Python.
What counts as solved
A public kernel and sweep at pinned versions that meets the bar on all 40 shapes, rerun by a second party.
How it gets verified
GPU benchmarkA kernel run and timed on stated hardware against a baseline.
Hardware: one H100 SXM 80 GB. Dtypes: BF16 inputs and output, FP32 accumulation. Shapes: 40 listed (M, N, K) triples published in the harness repo, covering decode (M 1 to 64), prefill (M 2048 to 16384) and non-multiples of 128. Baseline: cuBLAS (cublasLt) from CUDA 12.8 or later, exact version recorded, via torch.matmul with the version of PyTorch recorded. Metric: median TFLOP/s over at least 100 iterations after warmup, CUDA events, tuning time excluded but reported with a limit of 20 configurations per shape. Correctness: max relative error versus FP32 reference no more than 1.1x that of cuBLAS. Harness: public script at a pinned commit and Triton release.
Claims under verification
+ add a claimNo claimed results on the Verification board for this problem yet.
Where the work is
Compiler engineering and per-shape tuning, measured on H100.
Bottlenecks
ranked by votes- #1EngineeringCompiler support for Hopper asynchrony
TMA, WGMMA and warp specialisation must be selected automatically for the pipeline to reach peak.
Community grades
Sign in to add your own grade.