What Characteristics Make ChatGPT Effective for Software Issue Resolution? An Empirical Study of Task, Project, and Conversational Signals in GitHub Issues

Abstract

Conversational large-language models are extensively used for issue resolution tasks. However, not all developer-LLM conversations are useful for effective issue resolution. In this paper, we analyze 686 developer-ChatGPT conversations shared within GitHub issue threads to identify characteristics that make these conversations effective for issue resolution. First, we analyze the conversations and their corresponding issues to distinguish helpful from unhelpful conversations. We begin by categorizing the types of tasks developers seek help with to better understand the scenarios in which ChatGPT is most effective. Next, we examine a wide range of conversational, project, and issue-related metrics to uncover factors associated with helpful conversations. Finally, we identify common deficiencies in unhelpful ChatGPT responses to highlight areas that could inform the design of more effective developer-facing tools. We found that only 62% of the ChatGPT conversations were helpful for successful issue resolution. ChatGPT is most effective for code generation and tools/libraries/APIs recommendations, but struggles with code explanations. Helpful conversations tend to be shorter, more readable, and exhibit stronger semantic and linguistic alignment. Larger, more popular projects and more experienced developers benefit more from ChatGPT. At the issue level, ChatGPT performs best on simpler problems with limited developer activity and faster resolution, typically well-scoped tasks like compilation errors. The most common deficiencies in unhelpful ChatGPT responses include incorrect information and lack of comprehensiveness. Our findings have wide implications including guiding developers on effective interaction strategies for issue resolution, informing the development of tools or frameworks to support optimal prompt design, and providing insights on fine-tuning LLMs for issue resolution tasks.

Accepted for publication in Empirical Software Engineering (EMSE), 2025

References in corpus (11)

• The Programmer's Assistant: Conversational Interaction with a Large Language Model for Software Development
• Mark My Words! Linguistic Style Accommodation in Social Media
• Is Stack Overflow Obsolete? An Empirical Study of the Characteristics of ChatGPT Answers to Stack Overflow Questions
• Reducing hallucination in structured outputs via Retrieval-Augmented Generation
• Can We Use SE-specific Sentiment Analysis Tools in a Cross-Platform Setting?
• DevGPT: Studying Developer-ChatGPT Conversations
• Can ChatGPT Support Developers? An Empirical Evaluation of Large Language Models for Code Generation
• ChatGPT Code Detection: Techniques for Uncovering the Source of Code
• Human-interpretable clustering of short-text using large language models
• Evaluating Privacy Questions From Stack Overflow: Can ChatGPT Compete?
• BushraDBR: An Automatic Approach to Retrieving Duplicate Bug Reports