A hyperparameter-free training algorithm that beats Schedule-Free AdamW on AlgoPerf
Difficulty 42: in the first competition the self-tuning winner beat the NadamW baseline by only 0.035, and every other entry scored below it, so a hyperparameter-free method that works across all workloads is genuinely hard. Application value 44: tuning-free training saves whole sweeps. Leverage 22: a benchmark record.
The problem
Can a training method with no tuning at all beat Schedule-Free AdamW on the AlgoPerf v1.0 benchmark?
- 1AlgoPerf times training to target
- 2Self-tuning: one setting for everything
- 3v0.5 winner: Schedule-Free AdamW
- 4Beat it when rescored on v1.0
The yardstick. Neural networks are trained by an optimizer, the rule that turns derivatives of the loss into weight updates (examples: plain gradient descent, Adam, AdamW). Optimizer papers often claim speedups that vanish when the competitor is tuned as carefully. AlgoPerf, run by MLCommons, settles such claims by measuring wall-clock time to reach a fixed validation target on several fixed workloads (image classification, speech recognition, translation, graph and recommendation models) on fixed hardware, 4 NVIDIA A100 GPUs in version 1.0. Times across workloads are combined into one score between 0 and 1 using performance profiles: roughly, the share of workloads on which a method is close to the fastest one.
Path to understanding
foundations → problemThe research entries this problem rests on, in the order to read them. Each links to its explainer.
- Start with
- Step 2
- Adam, AdamW and the optimizer zooAlgorithm · host
- Then
- A hyperparameter-free training algorithm that beats Schedule-Free AdamW on AlgoPerfThe problem
State of the art
The first AlgoPerf competition (benchmark v0.5) received 18 submissions from 10 teams (Kasimbeg et al., ICLR 2025). Distributed Shampoo won the external-tuning ruleset and Schedule-Free AdamW won the self-tuning ruleset. On the v0.5 self-tuning leaderboard Schedule-Free AdamW scored 0.8542 against 0.8194 for the NadamW baseline, a small margin; the other self-tuning entries (sequential NadamW, learned optimizers, AdamG) scored lower than the baseline. The live leaderboard (version 0.6, last updated 2025-03-24) shows no scored self-tuning entries yet, and the benchmark README says the working group is focusing on the self-tuning leaderboard and plans v1.0 self-tuning baselines. Scores are only comparable within one benchmark version.
Realistic applications
Training algorithms that work without per-task tuning cut the number of full training runs needed to start a new project; the benchmark is the shared yardstick for claims of this kind.
What counts as solved
A submission merged and scored by the AlgoPerf working group on the self-tuning leaderboard of v1.0 (or a later version), above the Schedule-Free AdamW entry scored on the same version and above every other self-tuning entry at that time.
How it gets verified
ReproductionA training run, experiment or computation rerun by someone else.
Benchmark: AlgoPerf v1.0, self-tuning ruleset, from mlcommons/algorithmic-efficiency at the release tag the organisers score with; hardware 4x NVIDIA A100 40 GB as the v1.0 rules specify. Workloads: all base workloads in the v1.0 rules. Metric: the AlgoPerf benchmark score (normalised area under the performance profile of time-to-target), computed by the scoring code in mlcommons/submissions_algorithms. Baseline: Schedule-Free AdamW (Yang, Defazio, Mishchenko) self-tuning submission rescored on v1.0; the v0.5 numbers (0.8542 vs 0.8194 NadamW) are not comparable across versions. Threshold: strictly higher score; the repetition studies required by the rules (v0.6 and later) show the order holds. Constraint: a single hyperparameter configuration, no workload detection.
Claims under verification
+ add a claimNo claimed results on the Verification board for this problem yet.
Where the work is
Algorithm design plus careful engineering inside the benchmark rules.
Bottlenecks
ranked by votes- #1TheoryNo tuning allowed
A method must adapt its own step sizes across very different workloads within the timed budget.
Community grades
Sign in to add your own grade.
Connected problems
Progress
- Feb 20, 2025First competition analysedsource
Schedule-Free AdamW based submission won the self-tuning ruleset.