RepairBench: Leaderboard of Frontier Models for Program Repair
arXiv:2409.18952 · doi:10.1109/LLM4Code66737.2025.00006
Abstract
AI-driven program repair uses AI models to repair buggy software by producing patches. Rapid advancements in AI surely impact state-of-the-art performance of program repair. Yet, grasping this progress requires frequent and standardized evaluations. We propose RepairBench, a novel leaderboard for AI-driven program repair. The key characteristics of RepairBench are: 1) it is execution-based: all patches are compiled and executed against a test suite, 2) it assesses frontier models in a frequent and standardized way. RepairBench leverages two high-quality benchmarks, Defects4J and GitBug-Java, to evaluate frontier models against real-world program repair tasks. We publicly release the evaluation framework of RepairBench. We will update the leaderboard as new frontier models are released.
References in corpus (18)
- Evaluating Large Language Models Trained on Code
- Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
- DeepSeek-V3 Technical Report
- Less Training, More Repairing Please: Revisiting Automated Program Repair via Zero-shot Learning
- DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
- Bears: An Extensible Java Bug Benchmark for Automatic Program Repair Studies
- Qwen2.5-Coder Technical Report
- The Stack: 3 TB of permissively licensed source code
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions
- Agentless: Demystifying LLM-based Software Engineering Agents
- A Systematic Literature Review on Large Language Models for Automated Program Repair
- GitBug-Actions: Building Reproducible Bug-Fix Benchmarks with GitHub Actions
- CigaR: Cost-efficient Program Repair with LLMs
- Long Code Arena: a Set of Benchmarks for Long-Context Code Models
- HelpSteer2-Preference: Complementing Ratings with Preferences
- Are Large Language Models Memorizing Bug Benchmarks?
- The Fact Selection Problem in LLM-Based Program Repair