GPU Kernels & Systems

The kernels, compilers and communication layers that turn accelerator hardware into training and inference speed.

Sub-fields

GEMM & Attention KernelsMatrix-multiply and attention kernels tuned to run near the hardware's peak, measured in TFLOP/s.4 entries · 5 open problems
Kernel Languages & CompilersTriton, CUTLASS, TVM and the languages and compilers that generate fast kernels without hand-written assembly.2 entries · 3 open problems
Low-Precision NumericsFP8, FP4 and integer formats, and the scaling and rounding that keep low-precision training and inference accurate.2 entries · 3 open problems
Distributed Training & CommunicationCollectives, parallelism strategies and interconnects that spread one model across thousands of accelerators.2 entries · 2 open problems

Open problems, most leverage first

All open problems →

#VotesProblemWork mixDVLStatus

010LLM-written kernels that beat PyTorch on half of KernelBench Level 3 under hardened checksGPU benchmarkKernel autotuning and automated kernel generation1 link3 bottlenecks
385540OpenD 38V 55L 40▲ 0

Research

Filter in Research →

Low-Precision Numerics</> 3 reposTheoryEmerging

Microscaling (MX) and block-scaled 4-bit formats

Data formats that share one scale across a small block of narrow elements (MXFP8, MXFP6, MXFP4, MXINT8, and NVIDIA's NVFP4), implemented in Blackwell tensor cores.

2 problems0 dependents0 maintainers

Low-Precision Numerics</> 2 reposTheoryActive

FP8 formats and scaling recipes for training and inference

The two 8-bit floating-point encodings (E4M3 and E5M2), why tensors need scale factors, and the recipes (per-tensor, delayed, fine-grained blockwise) that make FP8 GEMMs usable for LLMs.

GPU Kernels & Systems

Sub-fields

Open problems, most leverage first

Research

Microscaling (MX) and block-scaled 4-bit formats

FP8 formats and scaling recipes for training and inference

GPU Kernels & Systems

Sub-fields

Open problems, most leverage first

Research

Microscaling (MX) and block-scaled 4-bit formats

FP8 formats and scaling recipes for training and inference

Kernel autotuning and automated kernel generation

Triton: a tile-level language and compiler for GPU kernels

Paged attention and KV-cache kernels

FlashAttention: IO-aware exact attention

Tiled GEMM on tensor cores (CUTLASS and CuTe)

The GPU memory hierarchy and the roofline model

Tensor, data and pipeline parallelism with compute-communication overlap

NCCL and collective communication: ring and tree all-reduce