A paged decode-attention kernel at 90% of achievable HBM bandwidth on H100
Difficulty 26: the ceiling is fixed by copy bandwidth and FlashMLA already reports about 90% on H800 in its best regime; the work is covering small, skewed and quantised batches in one kernel. Application value 62, the highest in the pack: decode dominates serving cost per token at very large deployed scale. Leverage 28: serving engines depend on it but little else does.
The problem
Can the step that produces each new word read the stored notes at nearly full memory speed in every situation?
- 1Each new word re-reads stored notes
- 2Little arithmetic, much memory reading
- 3Ceiling is plain copy speed
- 4Balance uneven requests over the chip
- 590% of copy speed everywhere
What is asked. When a language model writes text, it produces one token at a time. Each new token must attend to the stored keys and values of every earlier token, the KV cache. Serving engines store that cache in fixed-size blocks scattered through memory and found through a lookup table (paged attention; see that entry). The problem asks for one decode-attention kernel that reads the cache at 90% or more of the H100's practical memory speed, measured as the speed of a plain memory copy. It must do so in every regime: batches of 1 to 256 requests, contexts of 1k to 128k tokens, BF16 or FP8 caches, and batches where request lengths differ wildly.
Path to understanding
foundations → problemThe research entries this problem rests on, in the order to read them. Each links to its explainer.
- Start with
- Step 2
- Step 3
- Step 4
- Paged attention and KV-cache kernelsAlgorithm · host
- Then
- A paged decode-attention kernel at 90% of achievable HBM bandwidth on H100The problem
State of the art
Decode attention has very low arithmetic intensity, so its ceiling is HBM bandwidth. Production kernels approach that ceiling for large uniform batches: FlashMLA's README reported up to 3000 GB/s for memory-bound MLA decoding on H800 SXM5 (about 90% of the 3.35 TB/s peak), and FlashInfer and FlashAttention split-KV kernels target the same regime, but small batches with long contexts, skewed batches (one very long request among short ones), small query-head counts and quantised caches leave SMs idle or add reduction overhead. The curator did not find a primary source reporting a kernel at 90% of copy bandwidth across all those regimes.
Realistic applications
Decode time per token is a large part of serving latency and cost; a uniformly bandwidth-efficient kernel directly lowers cost per generated token.
What counts as solved
A public kernel and harness at a pinned commit showing at least 90% of the copy-benchmark bandwidth on every configuration, reproduced independently.
How it gets verified
GPU benchmarkA kernel run and timed on stated hardware against a baseline.
Hardware: one H100 SXM 80 GB. Shapes: batch 1, 8, 32, 128, 256; context 1k to 128k (cache must fit in 80 GB, configurations that do not fit are skipped and listed); head dim 128; 32 query heads with 32, 8 or 4 KV heads; block sizes 16 and 64; BF16 and FP8 E4M3 cache with per-head scales. Skewed batches: lognormal context lengths with stated parameters. Baselines: FlashInfer decode and FlashAttention split-KV with block table at pinned versions, plus a device-to-device copy kernel measured on the same card as the bandwidth reference. Metric: effective bytes read of K and V divided by kernel time, as % of the copy bandwidth, median of at least 100 iterations. Correctness: max absolute error versus an FP32 reference no larger than 2x the baselines. Harness: public script at a pinned commit.
Claims under verification
+ add a claimNo claimed results on the Verification board for this problem yet.
Where the work is
Mostly engineering against a roofline, with profiling-driven iteration.
Bottlenecks
ranked by votes- #1EngineeringLoad balance across SMs for skewed batches
One very long sequence among short ones leaves most SMs idle unless the split adapts to the length distribution.
Community grades
Sign in to add your own grade.
Connected problems
No connections yet.