RepairLLaMA: Efficient Representations and Fine-Tuned Adapters for Program Repair

Abstract

Automated Program Repair (APR) has evolved significantly with the advent of Large Language Models (LLMs). Fine-tuning LLMs for program repair is a recent avenue of research, with many dimensions which have not been explored. Existing work mostly fine-tune LLMs with naive code representations and does not scale to frontier models. To address this problem, we propose RepairLLaMA, a novel program repair approach that 1) identifies optimal code representations for APR with fine-tuned models, and 2) pioneers state-of-the-art parameter-efficient fine-tuning technique (PEFT) for program repair. This results in RepairLLaMA producing a highly effective `program repair adapter' for fixing bugs with AI. Our experiments demonstrate the validity of both concepts. First, fine-tuning adapters with program repair specific code representations enables the model to use meaningful repair signals and produce better patches. Second, parameter-efficient fine-tuning helps fine-tuning to converge and clearly contributes to the effectiveness of RepairLLaMA in fixing bugs outside the fine-tuning data distribution. Overall, RepairLLaMA correctly fixes 144 Defects4J v2, 109 HumanEval-Java, and 20 GitBug-Java bugs, outperforming all baselines.

Accepted to IEEE TSE

References in corpus (9)

• Practical Program Repair in the Era of Large Pre-trained Language Models
• Less Training, More Repairing Please: Revisiting Automated Program Repair via Zero-shot Learning
• No More Fine-Tuning? An Experimental Evaluation of Prompt Tuning in Code Intelligence
• Keep the Conversation Going: Fixing 162 out of 337 bugs for $0.42 each using ChatGPT
• Copiloting the Copilots: Fusing Large Language Models with Completion Engines for Automated Program Repair
• SelfAPR: Self-supervised Program Repair with Test Execution Diagnostics
• ITER: Iterative Neural Repair for Multi-Location Patches
• Revisiting the Plastic Surgery Hypothesis via Large Language Models
• Program Repair with Minimal Edits Using CodeT5

Cited by in corpus (2)

• Seeing is Fixing: Cross-Modal Reasoning with Multimodal LLMs for Visual Software Issue Fixing
• MultiMend: Multilingual Program Repair with Context Augmentation and Multi-Hunk Patch Generation