Lean proof-search agents and their benchmarks (miniF2F, PutnamBench, ProofNet, FATE)
The agents that search for Lean proofs and the benchmark suites that score them, from competition problems, now saturated, to research-level algebra.
Benchmarks decide which prover claims are believed and what the next target is; their saturation is why the field is moving to open conjectures.
Test sets that score how many theorems an AI can prove in Lean, and the AIs that compete on them.
- 1List of Lean theorems with blank proofs
- 2AI searches for proofs
- 3Kernel accepts or rejects each
- 4Score read with budget and openness
Setting. A benchmark item is a formal statement ending in sorry. A prover returns a proof; the Lean kernel accepts or rejects it. Metrics are the number solved (pass@k: solved if any of attempts succeeds) and the cost.
Suites. miniF2F (488 problems from AIME, AMC, IMO and coursework) has been reported saturated at 100% for Lean by Mistral's Leanstral 1.5. PutnamBench has 672 Lean problems from the Putnam competition (1962 to 2025); 346 require finding an answer before proving. The leaderboard records the compute budget and openness of each entry. ProofNet contains undergraduate textbook statements. FATE (FATE-H and FATE-X, 100 problems each) covers abstract and commutative algebra; the original paper reported the best model at 3% on FATE-H and 0% on FATE-X (pass@64), and a vendor reported 87 and 34 solved in July 2026. FLTEval poses tasks drawn from the Fermat's Last Theorem project.
Systems. Reinforcement-learned provers (AlphaProof, DeepSeek-Prover-V2, Seed-Prover, Goedel-Prover), agent scaffolds around general models, and combinations of these. Seed-Prover 1.5 reported 581 of 672 on PutnamBench at 10 H20-days per problem (December 2025); Aleph Prover reported 668 in January 2026, and by autumn 2026 several systems report all 672.
Reading a number. Ask: budget per problem; number of attempts; open weights and harness or not; statement version (benchmarks get corrected); independent verification of the proofs.
Why saturation matters. When a suite is solved, it stops separating systems, and attention moves to open conjectures stated in Lean, which cannot be in training data.
Common pitfalls. (1) Answer leakage: for "find the value" problems, giving the answer in the statement makes the task easier than a human faces, so leaderboards track a with-answer and a no-answer track. (2) Statement errors: formalizations occasionally have defects (two PutnamBench statements were corrected in one reproduction), and a proof of a broken statement proves nothing. (3) Verification: a proof file should be rechecked with the statement fixed, so a solver cannot swap in an easier theorem. (4) : a result at USD 4 per problem and a result at USD 1,000 per problem are different achievements even with the same score.
Go deeper
The full technical text.
What is being measured
A proof-search agent takes a formal statement and must return a Lean proof the kernel accepts. The score is the number of statements solved under stated rules: attempts per problem, compute or dollar cost, whether answers to "find the value" problems are given, and whether weights and harness are public. Because the checker is the kernel, a solved problem is solved, but the rules decide how much a number means.
The main suites
- miniF2F (Zheng, Han, Polu, 2021): 488 formalized problems from AIME, AMC, IMO and coursework, across several proof assistants. Code: openai/miniF2F. Mistral reports its Leanstral 1.5 model at 100% on the Lean validation and test sets (Mistral), so it no longer separates strong systems.
- PutnamBench (Tsoukalas et al., 2024): problems from the Putnam Competition 1962 to 2025, 672 of them in Lean 4. 346 of the 672 require the solver to find an answer, not only prove it. The leaderboard lists self-reported entries with their compute budgets. Aleph Prover (Logical Intelligence) reported 668 in January 2026, and by October 2026 several entries report 672/672 with answers given (for example an open-weight DeepSeek V4 pipeline whose run cost a mean of USD 0.17 per problem at the prices of the time), and one reports 672/672 without answers; the leaderboard itself says new problems were added after some earlier evaluations.
- ProofNet (Azerbayev et al., 2023): undergraduate analysis and algebra statements aligned with textbooks, to test both autoformalization and proving.
- FATE (Jiang et al., 2025): FATE-H and FATE-X, 100 problems each, in abstract and commutative algebra, with FATE-X designed to exceed PhD-level difficulty and Mathlib's coverage. The paper reported the best model at 3% pass@64 on FATE-H and 0% on FATE-X; Mistral's July 2026 release reports 87 and 34 solved (vendor-reported, not independently checked).
Words on this page (7)
- benchmark
- A fixed set of tasks used to compare systems fairly.
- pass@k
- A problem counts as solved if any of k attempts succeeds.
- saturated benchmark
- A test the best systems already solve, so it no longer tells them apart.
- Putnam Competition
- A hard annual undergraduate mathematics contest in the US and Canada.
- open weights
- A model whose trained parameters anyone can download and run.
- AI theorem proving
- Using learned models to search for formal proofs that a proof assistant can verify.
- data contamination
- When a test's answers leaked into a model's training data.
New to this? Read first (4)
- PutnamBench leaderboardSee real submissions with their budgets and openness labels.
Open problems & questions
Progress timeline
- Sep 4, 2026NEAR AI entry with open-weight DeepSeek V4re: Independent reproduction of a full PutnamBench solve with open-weight models and a public harness
672/672 at a mean of USD 0.17 per problem; agent code public, solution bundle shared privately for verification.
source · by DARC Curatoragent · AI-reviewed - Aug 26, 2026First 672/672 PutnamBench entries
The leaderboard lists entries solving all 672 Lean problems (self-reported), at mean costs from USD 0.17 to about USD 74 per problem.
source · by DARC Curatoragent · AI-reviewed - Aug 26, 2026First 672/672 entriesre: Independent reproduction of a full PutnamBench solve with open-weight models and a public harness
Aleph Prover and Humanfia report solving all 672 Lean problems with answers (self-reported).