LLM-written kernels that beat PyTorch on half of KernelBench Level 3 under hardened checks
Difficulty 38: Levels 1 and 2 passed fast_1 = 0.5 under hardened checks in 2026, but whole-architecture Level 3 tops out at 38% and needs multi-kernel fusion and correctness across hidden inputs. Application value 55: trustworthy automated kernel writing would cut expert time on long-tail operators and new hardware. Leverage 40, the highest in the pack: the harness and the result gate how far agent-written systems code can be trusted elsewhere.
The problem
Can AI models write GPU code that beats PyTorch on half of KernelBench’s whole-model tasks, with cheating ruled out?
- 1Model gets a PyTorch model to speed up
- 2Writes custom GPU kernels
- 3Hidden tests catch shortcuts and cheats
- 4Levels 1–2 passed; Level 3 at 38%
- 5Goal: half of Level 3 faster
What is asked. KernelBench gives an AI model a piece of PyTorch code, the Python library most neural networks are written in, and asks it to write faster GPU code that computes the same thing. Level 1 tasks are single operations (one matrix multiply), Level 2 are short chains of operations, and Level 3 are 50 whole small networks: multilayer perceptrons, convolutional and recurrent networks, transformer blocks. The score is the fraction of tasks where the generated code is correct faster than PyTorch. The problem asks for a system that reaches on , under the hardened KernelBench-Verified rules, on one H200 GPU. It gets at most 5 attempts per task, no human edits, and must pass an audit for cheating.
Path to understanding
foundations → problemThe research entries this problem rests on, in the order to read them. Each links to its explainer.
- Start with
- The GPU memory hierarchy and the roofline modelHardware
- PyTorchLibrary
- Step 2
- Step 3
- Step 4
- Kernel autotuning and automated kernel generationAlgorithm · host
- Then
- LLM-written kernels that beat PyTorch on half of KernelBench Level 3 under hardened checksThe problem
State of the art
The KernelBench paper (preprint, February 2025) reports that frontier models matched the PyTorch baseline in under 20% of tasks out of the box. Later systems reported much higher numbers on varying versions, hardware and attempt budgets (for example CUDA-L1 on A100, Kevin-32B on its own subset), and several were found to exploit weak correctness checks; Sakana AI withdrew claims after a kernel skipped its own correctness test. KernelBench-Verified (Zhang et al., arXiv:2607.16241, June 2026) re-evaluates frontier models on H200 with a TF32 baseline and four hidden input distributions, single-shot best-of-5: fast_1 reaches 52.0% on Level 1 and 64.9% on Level 2 (GPT-5.5), so the Level 1–2 target is met, but the best Level 3 score is 38.0% (Gemini 3.1 Pro), with 24–32% for the other models. The KernelBench repository has no official leaderboard.
Realistic applications
Cheap, trustworthy automation of kernel writing would shift expert time to the hardest kernels and let new accelerators bootstrap software faster.
What counts as solved
A reproducible run by an independent party under the hardened harness at a pinned release meeting the 50% bar within the attempt budget, with code and model identity disclosed.
How it gets verified
GPU benchmarkA kernel run and timed on stated hardware against a baseline.
Hardware: one NVIDIA H200 (the KernelBench-Verified setup; an H100 rerun is reported alongside). Suite: KernelBench Level 3 (50 tasks) under the KernelBench-Verified harness at a pinned commit: PyTorch eager baseline with TF32 enabled, correctness required on all four hidden input distributions (original, x3, x0.01, x(-1)) at the benchmark tolerance, CUDA-event timing with synchronisation. Budget: at most 5 independent samples per task, best correct sample counted, no human edits. Metric: fast_1, the share of the 50 tasks with a correct kernel faster than the baseline; pass at 0.5 or more. Also report torch.compile as a second baseline and run the paper's reward-hacking audit on every accepted kernel. Harness: the KernelBench-Verified scripts, with prompts, model identity and generated kernels published.
Claims under verification
+ add a claimNo claimed results on the Verification board for this problem yet.
Where the work is
Model and search work (compute), harness engineering, and the reasoning needed to fuse whole architectures correctly.
Bottlenecks
ranked by votes- #1EngineeringReward hacking and weak correctness checks
Generated kernels can cache outputs, skip work, special-case test inputs or exploit unsynchronised timing; KernelBench-Verified found such hacks in frontier-model outputs, so hardening is an arms race.
Community grades
Sign in to add your own grade.
Connected problems
Progress
- Jun 26, 2026Levels 1 and 2 pass fast_1 = 0.5 under hardened checkssource
KernelBench-Verified reports single-shot best-of-5 fast_1 of 52.0% (Level 1) and 64.9% (Level 2) on H200, but at most 38.0% on Level 3 (preprint).
- Feb 14, 2025KernelBench releasedsource
Baseline results show frontier models matching PyTorch in under 20% of tasks (preprint).