A Quantitative and Qualitative Evaluation of LLM-Based Explainable Fault Localization

Abstract

Fault Localization (FL), in which a developer seeks to identify which part of the code is malfunctioning and needs to be fixed, is a recurring challenge in debugging. To reduce developer burden, many automated FL techniques have been proposed. However, prior work has noted that existing techniques fail to provide rationales for the suggested locations, hindering developer adoption of these techniques. With this in mind, we propose AutoFL, a Large Language Model (LLM)-based FL technique that generates an explanation of the bug along with a suggested fault location. AutoFL prompts an LLM to use function calls to navigate a repository, so that it can effectively localize faults over a large software repository and overcome the limit of the LLM context length. Extensive experiments on 798 real-world bugs in Java and Python reveal AutoFL improves method-level acc@1 by up to 233.3% over baselines. Furthermore, developers were interviewed on their impression of AutoFL-generated explanations, showing that developers generally liked the natural language explanations of AutoFL, and that they preferred reading a few, high-quality explanations instead of many.

Accepted to ACM International Conference on the Foundations of Software Engineering (FSE 2024)

References in corpus (13)

• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Large Language Models are Zero-Shot Reasoners
• Self-Consistency Improves Chain of Thought Reasoning in Language Models
• ReAct: Synergizing Reasoning and Acting in Language Models
• Reflexion: Language Agents with Verbal Reinforcement Learning
• HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face
• BugsInPy: A Database of Existing Bugs in Python Programs to Enable Controlled Testing and Debugging Studies
• A Quantitative and Qualitative Evaluation of LLM-Based Explainable Fault Localization
• Conversational Automated Program Repair
• Explaining Software Bugs Leveraging Code Structures in Neural Machine Translation
• Large Language Models in Fault Localisation
• Explainable Automated Debugging via Large Language Model-driven Scientific Debugging

Cited by in corpus (3)

• A Quantitative and Qualitative Evaluation of LLM-Based Explainable Fault Localization
• ChatDBG: Augmenting Debugging with Large Language Models
• A Systematic Literature Review of the Use of GenAI Assistants for Code Comprehension: Implications for Computing Education Research and Practice