Half of FATE-X: an open-weight prover solving 50 of 100 PhD-level algebra statements
Difficulty 35: a vendor-reported jump from 0 to 34 of 100 in eight months shows a path, but the remaining statements need algebra that Mathlib lacks to be built first, and the open-weight, fixed-budget, reproduced conditions are strict; it sits just below the Mathlib-integrated FLT formalization (38), which is a far larger engineering effort, and well above the PutnamBench rerun (8). Application value 28: a prover at this level would help fill Mathlib's algebra gaps and check graduate algebra, just below the FLT library (29). Leverage 26: a milestone for the claim that provers handle research-level material, below the kernel-conformance proof (42).
The problem
Can a downloadable AI prove half of 100 PhD-level algebra statements in Lean?
- 1100 hard algebra statements in Lean
- 2Best model in 2025 scored 0
- 3A vendor reports 34 in 2026
- 4Target: 50, independently reproduced
The benchmark. Abstract algebra studies structures such as groups and rings (see the Mathlib hierarchy entry); commutative algebra studies rings like the polynomials, and is the language of algebraic geometry. FATE (Formal Algebra Theorem Evaluation; Jiang et al., 2025) has FATE-M (150), FATE-H (100) and FATE-X (100) statements in abstract and commutative algebra. FATE-X contains problems at PhD qualifying exam level and beyond, from textbooks, exams and research literature, with minimal supporting definitions before the statement. Each file has one sorry.
Results so far. The paper's best model reached 3% (pass@64) on FATE-H and 0% on FATE-X,. In July 2026 Mistral reported Leanstral 1.5 (119B parameters, 6B active, Apache-2.0) at 87 on FATE-H and 34 on FATE-X, with the release page giving both percentages and counts and no stated budget for FATE. FATE has 100 problems per set, so 34 would be 34%.
Path to understanding
foundations → problemThe research entries this problem rests on, in the order to read them. Each links to its explainer.
- Start with
- Lean 4 & MathlibLibrary
- Step 2
- Machine-assisted theorem provingFrontier
- Step 3
- Then
- Half of FATE-X: an open-weight prover solving 50 of 100 PhD-level algebra statementsThe problem
State of the art
In the November 2025 paper the best evaluated model reached 0% pass@64 on FATE-X (and 3% on FATE-H) (arXiv:2511.02872). In July 2026 Mistral reported Leanstral 1.5, an Apache-2.0 open-weight model with 119B parameters (6B active), at 87 problems on FATE-H and 34 problems on FATE-X, for 34% of FATE-X; the release page gives the figures once as percentages and once as problem counts and does not state the pass@k or budget for FATE (vendor-reported, not independently reproduced) (Mistral). Qualitatively, FATE-X statements often need definitions or results outside Mathlib's coverage, so a solver must also build supporting theory.
Realistic applications
FATE-X measures a skill the field wants: proving graduate algebra in a library that is missing parts of it. A prover that clears half would be useful for filling Mathlib's algebra gaps and for checking research-level algebra papers, and it shows whether competition-style progress carries over to research mathematics.
What counts as solved
A public report with model weights, harness, pass@k, per-problem token or dollar cost, the FATE commit and the Lean and Mathlib versions, giving at least 50 of 100 kernel-checked solutions on unmodified statements, and an independent reproduction of that count by a second party. Reproducing the vendor's 34 of 100 first is a necessary stepping stone.
How it gets verified
ReproductionA training run, experiment or computation rerun by someone else.
Benchmark: FATE-X, 100 Lean files, each with one sorry and minimal supporting definitions, pinned by commit. Metric: number of the 100 files for which the submitted proof is accepted by the kernel against the unmodified statement (Comparator or SafeVerify, which Mistral publishes as a fork, LeanstralSafeVerify); the threshold is at least 50. Rules to state: pass@k (k at most 8, matching the vendor's PutnamBench setting), model weights public, harness public, and the per-problem token budget. Variance: report the solved set over at least three independent seeds and use the intersection or the median, not the best run. The first milestone is reproducing the vendor's claimed 34, with the Lean and Mathlib versions the benchmark was built for.
Claims under verification
+ add a claimNo claimed results on the Verification board for this problem yet.
Where the work is
Progress comes from better provers and harnesses, and from building the supporting definitions and lemmas that FATE-X statements need in Lean.
Bottlenecks
ranked by votes- #1TheoryMathematics beyond Mathlib's coverage
FATE-X was designed to exceed Mathlib's coverage; solvers must prove auxiliary theory before the target statement.
Community grades
Sign in to add your own grade.
Connected problems
- This problem is related to Independent reproduction of a full PutnamBench solve with open-weight models and a public harness