An autotuner that finds configurations within 2% of the best in 20 trials
Difficulty 41: sample-efficient tuning that generalises to unseen shapes and a second GPU is an open machine-learning-for-systems question, harder than any single-kernel parity task here (20 to 34). Application value 39: removes tuning latency and shipped tables in serving engines. Leverage 24: benefits several kernel families; few results hinge on it.
The problem
Can a tuner find nearly the fastest GPU settings for a new size after trying only 20?
- 1Best settings change per size and chip
- 2Exhaustive search is too slow
- 3Learn a model that predicts fast settings
- 4Test on unseen sizes with 20 trials
- 5Near-best without long tuning
What is asked. A fast GPU kernel has settings: tile sizes, how many stages of data are loaded ahead, how many groups of threads share a tile, and the order tiles are visited. The best settings change with the matrix shape and the GPU model. A typical search space has to combinations per shape, and timing one combination means compiling and running it. The problem asks for a that, for a shape it has never seen, tries at most 20 settings and returns one whose speed is at least 98% of the best found by trying everything. That is required on average over 200 new shapes, and at least 93% on the worst tenth. It must work for matrix-multiply and attention kernels, on the H100 and a second named GPU.
Built from
earlier problems → this oneThe problems this one is built from: progress or tools it needs (builds on), special cases that come first (generalizes), and weaker results on the way (implies). On the right, what builds on it.
- Start with
- A Triton BF16 GEMM within 3% of cuBLAS across an LLM shape sweep on H100Kernel Languages & Compilers · D27
- Then
- An autotuner that finds configurations within 2% of the best in 20 trialsThis problem
Path to understanding
foundations → problemThe research entries this problem rests on, in the order to read them. Each links to its explainer.
- Start with
- The GPU memory hierarchy and the roofline modelHardware
- PyTorchLibrary
- Step 2
- Step 3
- Step 4
- Kernel autotuning and automated kernel generationAlgorithm · host
- Then
- An autotuner that finds configurations within 2% of the best in 20 trialsThe problem
State of the art
Learned cost models reduce the number of trials compared with random or grid search in the TVM line of work, and Triton's autotune relies on user-specified candidate lists with a full benchmark of each. Published numbers for sample efficiency depend on search spaces, hardware and shapes, so no cross-system 20-trial figure at 98% is established. Transfer of a tuner trained on one GPU generation to the next is also open.
Realistic applications
Serving engines see many dynamic shapes; cheap, near-optimal configuration selection removes both tuning latency and the need to ship large tables.
What counts as solved
A public tuner, search-space definition and exhaustively tuned ground-truth dataset showing the bounds on the held-out set, reproduced independently.
How it gets verified
GPU benchmarkA kernel run and timed on stated hardware against a baseline.
Hardware: H100 SXM 80 GB and one other named GPU (for example A100 80 GB or B200). Kernels: Triton BF16 GEMM and a Triton causal attention forward with the search space published in the harness repo. Ground truth: exhaustive timing of every configuration for 200 held-out shapes (shapes published after tuner freeze). Metric: for each shape, throughput of the tuner's selected configuration divided by the best exhaustive throughput, after at most 20 timed trials (compile time reported separately). Correctness: every candidate must pass the kernel's reference check. Harness: public tuner and evaluation script at a pinned commit.
Claims under verification
+ add a claimNo claimed results on the Verification board for this problem yet.
Where the work is
A learning and search problem needing exhaustive timing data (compute) and some engineering of the search space.
Bottlenecks
ranked by votes- #1TheoryGeneralisation across shapes and GPU generations
Cost models trained on one GPU transfer poorly to another and to shapes outside the training distribution.
Community grades
Sign in to add your own grade.