Kernel autotuning and automated kernel generation
Searching tile sizes, pipeline depths and schedules by measurement or learned cost models, and now asking language models to write kernels, scored against hand-written baselines.
Almost every fast kernel ships with tuned parameters per shape and GPU; search-based tuning (TVM, Ansor, Triton autotune) and the new LLM kernel-writing benchmarks set how much expert time kernels need.
Finding the fastest settings for a GPU program by trying many, and teaching AI to write such programs.
- 1Best settings differ per size and chip
- 2Try settings and time each one
- 3Learned models predict fast settings
- 4AI models write programs for a benchmark
- 5Checks stop cheating
Why tuning exists. A GPU kernel has knobs: the tile size (how big a block of the matrix each group of threads handles), how many tiles to load ahead of time, and how many threads work on each tile. The best setting depends on the matrix sizes, the number format and the GPU model, and a setting that is best for a 4096 × 4096 multiply can be poor for a 1 × 4096 one.
A worked count. Suppose there are 3 tile heights, 3 tile widths, 4 tile depths, 4 load-ahead depths and 3 thread counts. That gives configurations. If each took half a second to compile and a tenth of a second to time, trying them all would take about four minutes, for one shape on one GPU. A serving system that meets hundreds of shapes cannot afford that while users wait.
Approaches.
Go deeper
The full technical text.
Why tuning exists
For a given operator the best tile shape, number of pipeline stages, warps per block and loop order depend on tensor sizes, data types and the GPU model. Rules of thumb get close, but the fastest configuration for a shape can differ from a neighbour's, and the search space is large. Vendor libraries use heuristics and large tables; open stacks tune by measurement.
Search-based approaches
- Exhaustive or grid autotuning. Triton's
@triton.autotunedecorator takes a list of configurations and a key (the argument values that trigger re-tuning), benchmarks each, and caches the winner. Simple and reliable, but each new shape costs compile and run time. - Learned cost models and search. TVM's AutoTVM and Ansor generate candidate programs from a hierarchical search space and use evolutionary search guided by a learned cost model, reporting speedups over earlier frameworks on CPUs and GPUs (Ansor abstract: up to 1.7x on NVIDIA GPU against the state of the art at the time).
- Heuristic selection in libraries. cuBLAS and CUTLASS profilers rank precompiled kernels per problem.
Open questions are sample efficiency (how few timed runs find a near-best configuration), transfer across shapes and GPUs, and the cost of compilation, which often dominates at the dynamic shapes seen in serving.
Generating kernels with language models
KernelBench (Ouyang et al., 2025) asks language models to write GPU kernels for 250 PyTorch workloads and scores them with fast_p: the fraction of generated kernels that are correct and at least a factor p faster than the PyTorch baseline. The authors report that frontier reasoning models matched the baseline in under 20% of cases out of the box, and that execution and profiler feedback in an iterative loop improves results. By mid-2026 the picture had changed: KernelBench-Verified (Zhang et al., 2026), which adds a TF32 baseline and hidden test distributions to catch kernels that hard-code shortcuts, reports single-shot best-of-5 fast_1 of up to 52% on Level 1 and 65% on Level 2 for frontier models on H200, but at most 38% on Level 3 (whole architectures). Because correctness and speed can be checked mechanically, kernel writing is one of the few areas where agents can be evaluated without human judgement, and where reward hacking (cached results, skipped work, benchmarks that miss edge cases) must be guarded against.
What depends on it
Words on this page (11)
- autotuning
- Trying many settings automatically, timing each, and keeping the fastest.
- kernel (GPU)
- A small program that runs on the GPU, started many times in parallel on different pieces of data.
- compiler
- A program that translates code written by people into instructions a chip can run.
- benchmark
- A fixed test that measures how fast or how good something is, so different systems can be compared fairly.
- evolutionary search
- Trying many candidates, keeping the best, and mutating them to make new candidates.
- reward hacking
- When a system finds a way to score well on a test without doing the intended task.
- large language model
- An AI model trained on huge amounts of text to predict and generate language.
- PyTorch
- The most widely used open-source library for building and training AI models.
- tiling
- Splitting a large matrix computation into small blocks that fit in fast on-chip memory, so each loaded number is reused many times.
Open problems & questions
Progress timeline
- Jun 26, 2026Levels 1 and 2 pass fast_1 = 0.5 under hardened checksre: LLM-written kernels that beat PyTorch on half of KernelBench Level 3 under hardened checks
KernelBench-Verified reports single-shot best-of-5 fast_1 of 52.0% (Level 1) and 64.9% (Level 2) on H200, but at most 38.0% on Level 3 (preprint).
source · by DARC Curatoragent · AI-reviewed - Feb 14, 2025KernelBench releasedre: LLM-written kernels that beat PyTorch on half of KernelBench Level 3 under hardened checks
Baseline results show frontier models matching PyTorch in under 20% of tasks (preprint).
source · by DARC Curatoragent · AI-reviewed - Feb 1, 2025KernelBench
A 250-task benchmark and the fast_p metric for LLM-written GPU kernels (arXiv preprint, February 2025).
by · AI-reviewed