Beyond Compilation: Evaluating Faithful Natural-Language-to-Lean Statement Formalization
Lean verifies that a generated declaration is well typed, but cannot check whether it states what the user intended. We measure how far compilation overstates faithfulness in statement autoformalization without canonical Lean targets. Our criterion requires Lean compilation and agreement of two LLM judges from different model families. On an independent random audit, it agrees with the human majority on 91.5% of cases; it rarely accepts an unfaithful output but rejects some faithful ones. Across eight systems on 227 university-level statements, the gap between compiling and accepted outputs ranges from 1.3 to 29.5 percentage points: our tool-augmented agent compiles 87.2% of the statement...
| Authors | Ke Zhang, Patricio Gallardo Candela, Sudhir Murthy et al. |
|---|---|
| Primary category | cs.AI |
| Published | 30 June 2026 |
| Last updated at source | 15 October 2026 |
Popularity, last 4 days
0 now. Counts as reported by the source, snapshotted daily by Silytics.
Connected in the knowledge graph
- grenadeComing/beyond-compilation-benchmarkDataset · cited by
Recent changes
- 19 Oct 2026 Description: Lean verifies that a generated declaration is well typed, but not that it states what the user intended. For statement autoformalization without canonical Lean targets, we study two questions: how far an LLM-based semantic criterion can be → Lean verifies that a generated declaration is well typed, but cannot check whether it states what the user intended. We measure how far compilation overstates faithfulness in statement autoformalization without canonical Lean targets. Our c
Changes detected by Silytics between refreshes of the source.
Models and datasets referencing this paper
Silytics shows public metadata from arXiv and links to the original. Information may be incomplete or out of date. Last refreshed 20 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.