FP8 attention that keeps FlashAttention-3 throughput at BF16-level accuracy
Difficulty 40, highest of the attention kernels: it needs numerics research (outlier-robust scaling of Q and K on real activations) on top of kernel work, comparable to the parked FP8 accumulator-width question (35.5). Application value 48: would make FP8 attention usable where BF16 accuracy is required. Leverage 27: shapes FP8 inference stacks without gating other results.
The problem
Can attention run in 8-bit numbers at full speed without losing accuracy on real model data?
- 18-bit attention is fast but less accurate
- 2Real models have outlier channels
- 3Rotate and scale before rounding
- 4Test on captured real activations
- 5Near-8-bit speed, near-16-bit accuracy
What is asked. Running attention in FP8 (8-bit floating point) roughly doubles the matrix-multiply rate on an H100 compared with 16-bit BF16, but each number keeps only about one significant decimal digit. FlashAttention-3 already has an FP8 forward kernel. The problem asks for an FP8 attention kernel that keeps at least 95% of FlashAttention-3's FP8 speed (head dimension 128, 8192 tokens). Its error, measured against a 32-bit reference, must be no more than twice that of FlashAttention-3 in BF16. The test inputs are not random numbers but query, key and value tensors captured from named open-weights language models.
Path to understanding
foundations → problemThe research entries this problem rests on, in the order to read them. Each links to its explainer.
- Start with
- Step 2
- Step 3
- FlashAttention: IO-aware exact attentionAlgorithm · host
- Then
- FP8 attention that keeps FlashAttention-3 throughput at BF16-level accuracyThe problem
State of the art
FlashAttention-3's abstract reports FP8 forward close to 1.2 PFLOP/s and 2.6x lower numerical error than a baseline FP8 attention, using block quantisation and incoherent processing (random orthogonal transforms to spread outliers). The error remains larger than BF16 attention, and that statement is about random Gaussian-like inputs plus an outlier-injected test; behaviour on activations captured from specific production models is an open measurement question. FP8 backward attention is not covered by the FlashAttention-3 FP8 results.
Realistic applications
FP8 attention would roughly double attention tensor-core throughput on Hopper and relieve KV-cache bandwidth when K and V are stored in FP8.
What counts as solved
A public kernel and error report from captured activations of the listed models that meets both bounds, rerun by a second party.
How it gets verified
GPU benchmarkA kernel run and timed on stated hardware against a baseline.
Hardware: one H100 SXM 80 GB. Dtypes: Q, K, V in E4M3, FP32 softmax statistics, BF16 output. Shapes: head dim 128, sequence 8192, causal and non-causal, 32 heads, total tokens near 16384. Inputs: Q, K, V activations from layer 0, 15 and last layer of two named open-weights LLMs on a fixed 8k-token text sample, plus the synthetic outlier test of the FlashAttention-3 paper. Baselines: FlashAttention-3 FP8 and BF16 forward at a pinned commit. Metric: TFLOP/s median over at least 100 iterations; RMSE and max error versus an FP32 reference. Correctness bound as in the statement. Harness: public script and captured tensors (or capture script) at a pinned commit.
Claims under verification
+ add a claimNo claimed results on the Verification board for this problem yet.
Where the work is
A mix of numerics design and kernel work, with an empirical component from capturing activations of real models.
Bottlenecks
ranked by votes- #1TheoryOutlier-robust quantisation of Q and K
Attention scores are sensitive to outliers in a few channels; transforms spread outliers at some cost, and error bounds for softmax after quantisation are loose.
Community grades
Sign in to add your own grade.