Keep the Conversation Going: Fixing 162 out of 337 bugs for $0.42 each using ChatGPT

Abstract

Automated Program Repair (APR) aims to automatically generate patches for buggy programs. Recent APR work has been focused on leveraging modern Large Language Models (LLMs) to directly generate patches for APR. Such LLM-based APR tools work by first constructing an input prompt built using the original buggy code and then queries the LLM to generate patches. While the LLM-based APR tools are able to achieve state-of-the-art results, it still follows the classic Generate and Validate repair paradigm of first generating lots of patches and then validating each one afterwards. This not only leads to many repeated patches that are incorrect but also miss the crucial information in test failures as well as in plausible patches. To address these limitations, we propose ChatRepair, the first fully automated conversation-driven APR approach that interleaves patch generation with instant feedback to perform APR in a conversational style. ChatRepair first feeds the LLM with relevant test failure information to start with, and then learns from both failures and successes of earlier patching attempts of the same bug for more powerful APR. For earlier patches that failed to pass all tests, we combine the incorrect patches with their corresponding relevant test failure information to construct a new prompt for the LLM to generate the next patch. In this way, we can avoid making the same mistakes. For earlier patches that passed all the tests, we further ask the LLM to generate alternative variations of the original plausible patches. In this way, we can further build on and learn from earlier successes to generate more plausible patches to increase the chance of having correct patches. While our approach is general, we implement ChatRepair using state-of-the-art dialogue-based LLM -- ChatGPT. By calculating the cost of accessing ChatGPT, we can fix 162 out of 337 bugs for $0.42 each!

References in corpus (9)

• Training language models to follow instructions with human feedback
• SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repair
• Practical Program Repair in the Era of Large Pre-trained Language Models
• TBar: Revisiting Template-based Automated Program Repair
• Automatic Repair of Real Bugs in Java: A Large-Scale Experiment on the Defects4J Dataset
• Less Training, More Repairing Please: Revisiting Automated Program Repair via Zero-shot Learning
• FixMiner: Mining Relevant Fix Patterns for Automated Program Repair
• Automatic Repair of Buggy If Conditions and Missing Preconditions with SMT
• An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation

Cited by in corpus (10)

• Harnessing the Power of LLM to Support Binary Taint Analysis
• RepairLLaMA: Efficient Representations and Fine-Tuned Adapters for Program Repair
• Enhancing Genetic Improvement Mutations Using Large Language Models
• Prompt engineering for bibliographic web-scraping
• Deep Learning-Based Identification of Inconsistent Method Names: How Far Are We?
• Automated Testing of the GUI of a Real-Life Engineering Software using Large Language Models
• Hierarchical Knowledge Injection for Improving LLM-based Program Repair
• Seeing is Fixing: Cross-Modal Reasoning with Multimodal LLMs for Visual Software Issue Fixing
• Show Me Why It's Correct: Saving 1/3 of Debugging Time in Program Repair with Interactive Runtime Comparison
• MultiMend: Multilingual Program Repair with Context Augmentation and Multi-Hunk Patch Generation