Synergizing LLMs and Knowledge Graphs: A Novel Approach to Software Repository-Related Question Answering

Abstract

Software repositories contain valuable information for understanding the development process. However, extracting insights from repository data is time-consuming and requires technical expertise. While software engineering chatbots support natural language interactions with repositories, chatbots struggle to understand questions beyond their trained intents and to accurately retrieve the relevant data. This study aims to improve the accuracy of LLM-based chatbots in answering repository-related questions by augmenting them with knowledge graphs. We use a two-step approach: constructing a knowledge graph from repository data, and synergizing the knowledge graph with an LLM to handle natural language questions and answers. We curated 150 questions of varying complexity and evaluated the approach on five popular open-source projects. Our initial results revealed the limitations of the approach, with most errors due to the reasoning ability of the LLM. We therefore applied few-shot chain-of-thought prompting, which improved accuracy to 84%. We also compared against baselines (MSRBot and GPT-4o-search-preview), and our approach performed significantly better. In a task-based user study with 20 participants, users completed more tasks correctly and in less time with our approach, and they reported that it was useful. Our findings demonstrate that LLMs and knowledge graphs are a viable solution for making repository data accessible.

Submitted to ACM Transactions on Software Engineering and Methodology for review

References in corpus (20)

• A Survey on Knowledge Graphs: Representation, Acquisition and Applications
• Knowledge Graphs
• Unifying Large Language Models and Knowledge Graphs: A Roadmap
• What's in a GitHub Star? Understanding Repository Starring Practices in a Social Coding Platform
• The Impact of AI on Developer Productivity: Evidence from GitHub Copilot
• A Comparison of Natural Language Understanding Platforms for Chatbots in Software Engineering
• MSRBot: Using Bots to Answer Questions from Software Repositories
• Large Language Models are not Fair Evaluators
• CodeT5+: Open Code Large Language Models for Code Understanding and Generation
• BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions
• Using Large Language Models to Generate, Validate, and Apply User Intent Taxonomies
• Alibaba LingmaAgent: Improving Automated Issue Resolution via Comprehensive Repository Exploration
• Making Large Language Models Better Reasoners with Alignment
• Harnessing Large Language Models for Knowledge Graph Question Answering via Adaptive Multi-Aspect Retrieval-Augmentation
• GraphOTTER: Evolving LLM-based Graph Reasoning for Complex Table Question Answering
• GLoRe: When, Where, and How to Improve LLM Reasoning via Global and Local Refinements
• LLM-ARC: Enhancing LLMs with an Automated Reasoning Critic
• Can LLM Graph Reasoning Generalize beyond Pattern Memorization?
• Bridging Design and Development with Automated Declarative UI Code Generation
• Large Language Models, Knowledge Graphs and Search Engines: A Crossroads for Answering Users' Questions