The NanoGPT speedrun in under 30 seconds on 8×H100
Difficulty 23: a fixed 30-second target, about 25% below the current 40-second record, on a pipeline that dropped 46% in its last record; engineering on a heavily optimised baseline. Application value 16 and leverage 12, the lowest here: gains at 124M parameters on one node often do not transfer, though Muon came out of this testbed.
The problem
Can a small GPT-2-style model reach the speedrun's target loss in under 30 seconds on 8 H100 GPUs?
- 1Same text, same 8 H100s
- 2Reach validation loss 3.28
- 345 minutes in 2024, 40 s now
- 4Target: under 30 seconds
The game. The NanoGPT speedrun is an open competition: train a small language model, starting from random weights, until its validation loss (average cross-entropy, a measure of how surprised the model is by held-out text) on the FineWeb dataset reaches 3.28, as fast as possible on one machine with 8 NVIDIA H100 GPUs. The training text streams are fixed and may not be changed, and extra compiler flags are banned. Because two runs with different random seeds end at slightly different losses, a record needs enough runs to show statistically that the average is at most 3.28, with a p-value below 0.01 (the chance that luck alone explains the result is under 1%).
The 3.28 target comes from a careful GPT-2-small reproduction that took 45 minutes. By record 92 (30 August 2026) the time was 0.665 minutes, about 40 seconds; the two records before it were 1.126 and 1.13 minutes. That last jump combined many changes: sampling a subset of output words early in training, 8-bit floating-point (FP8) arithmetic in the feed-forward layers, a very large hashed table of word-pair and word-triple embeddings, a new optimizer stack, fewer layers with mixed-width attention, and capturing the whole training step as one replayable GPU program. The Muon optimizer was first developed here.
Path to understanding
foundations → problemThe research entries this problem rests on, in the order to read them. Each links to its explainer.
- Start with
- Step 2
- Adam, AdamW and the optimizer zooAlgorithm · host
- Then
- The NanoGPT speedrun in under 30 seconds on 8×H100The problem
State of the art
On 2026-10-17 the repository's record table ends at record 92: 0.665 minutes (about 40 seconds), dated 2026-08-30, combining sampled softmax early in training, FP8 MLP forward and backward passes, an 84.6M-row hashed n-gram embedding table, the ANVIL optimizer stack, depth reduction with mixed-width attention, and CUDA-graph capture of the training step. The previous records were 1.126 minutes (record 91, 2026-08-06) and 1.13 minutes (record 90, 2026-08-03); the llm.c GPT-2 baseline that set the 3.28 target took 45 minutes. Progress comes in irregular jumps, so the README table is the live reference.
Realistic applications
The speedrun is an open testbed where optimizer, architecture and systems ideas are checked on identical hardware against a strong baseline; Muon came out of it. Gains at 124M-parameter scale need not transfer to larger runs (see the linked problem on optimizer speedups at 1B+).
What counts as solved
A record accepted into the repository's main-track table at under 30 seconds, with the logs the rules require, or an equivalent independent run under the same rules and hardware that the maintainers confirm.
How it gets verified
GPU benchmarkA kernel run and timed on stated hardware against a baseline.
Hardware: one node of 8x NVIDIA H100 (SXM, NVLink), the repository's reference setup; driver, CUDA and PyTorch versions recorded. Task: the main-track train_gpt.py of KellerJordan/modded-nanogpt at the submitted commit, unchanged FineWeb train and validation token streams. Baseline: record 92 (0.665 min, 2026-08-30) rerun on the same node, since the rules require beating the prior record on the same hardware. Metric: timed training wall-clock as the script logs it (compilation excluded, as in the repository), mean over the submitted runs; pass if under 30.0 s. Correctness: mean validation loss ≤ 3.28 at p < 0.01 over the run logs (the repository's one-sided test), no extra torch.compile or inductor flags.
Claims under verification
+ add a claimNo claimed results on the Verification board for this problem yet.
Where the work is
Mostly kernel, precision and optimizer engineering.
Bottlenecks
ranked by votes- #1EngineeringDiminishing headroom on one node
Many obvious gains are taken; new ones need low-level kernel and numerics work.
Community grades
Sign in to add your own grade.
Connected problems
Progress
- Aug 30, 2026Record 92: about 40 secondssource
The repository table lists record 92 at 0.665 minutes by @DevenPzak (pull request 360).