Large Language Models for Fault Localization: An Empirical Study

Abstract

Large Language Models (LLMs) have demonstrated strong performance on code-related tasks, particularly in automated program repair. However, repair effectiveness often depends on accurate upstream fault localization, while the statement-level fault localization capability of LLMs remains insufficiently evaluated. This paper presents a systematic empirical study of LLMs for statement-level fault localization. We evaluate four representative LLMs, including two open-weight models, Qwen2.5-Coder-32B-Instruct and DeepSeek-V3, and two closed-source models, GPT-4.1 mini and Gemini-2.5-Flash, on HumanEval-Java and Defects4J. The evaluation covers different input contexts and prompt strategies, including Zero-shot, Few-shot, and Chain-of-Thought prompting. We further assess model performance from three complementary perspectives: Exact Match, Partial Match, and output consistency, and compare LLMs with representative non-LLM baselines, including PMD and LineDef, under the same source-code-only input setting. In addition, we analyze practical efficiency and cost in terms of end-to-end response time and token-based API cost. The results show that bug report context improves observed fault localization performance on Defects4J; Few-shot prompting improves performance in some cases but does not yield consistent gains; and Chain-of-Thought prompting shows mixed effects across models. Overall, this study reveals the strengths, limitations, and practical trade-offs of LLMs in statement-level fault localization, providing empirical evidence for model selection and application in software engineering practice.