A Case Study on the Effectiveness of LLMs in Verification with Proof Assistants

Abstract

Large language models (LLMs) can potentially help with verification using proof assistants by automating proofs. However, it is unclear how effective LLMs are in this task. In this paper, we perform a case study based on two mature Rocq projects: the hs-to-coq tool and Verdi. We evaluate the effectiveness of LLMs in generating proofs by both quantitative and qualitative analysis. Our study finds that: (1) external dependencies and context in the same source file can significantly help proof generation; (2) LLMs perform great on small proofs but can also generate large proofs; (3) LLMs perform differently on different verification projects; and (4) LLMs can generate concise and smart proofs, apply classical techniques to new definitions, but can also make odd mistakes.

Accepted by LMPL 2025

References in corpus (12)

• A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions
• Total Haskell is Reasonable Coq
• Passport: Improving Automated Formal Verification Using Identifiers
• ProofNet: Autoformalizing and Formally Proving Undergraduate-Level Mathematics
• The Tactician (extended version): A Seamless, Interactive Tactic Learner and Prover for Coq
• Cobblestone: A Divide-and-Conquer Approach for Automating Formal Verification
• QEDCartographer: Automating Formal Verification Using Reward-Free Reinforcement Learning
• CoqPilot, a plugin for LLM-based generation of proofs
• miniCodeProps: a Minimal Benchmark for Proving Code Properties
• Getting More out of Large Language Models for Proofs
• VerifyThisBench: Generating Code, Specifications, and Proofs All at Once
• HybridProver: Augmenting Theorem Proving with LLM-Driven Proof Synthesis and Refinement