Neural Bug Finding: A Study of Opportunities and Challenges
arXiv:1906.00307
Abstract
Static analysis is one of the most widely adopted techniques to find software bugs before code is put in production. Designing and implementing effective and efficient static analyses is difficult and requires high expertise, which results in only a few experts able to write such analyses. This paper explores the opportunities and challenges of an alternative way of creating static bug detectors: neural bug finding. The basic idea is to formulate bug detection as a classification problem, and to address this problem with neural networks trained on examples of buggy and non-buggy code. We systematically study the effectiveness of this approach based on code examples labeled by a state-of-the-art, static bug detector. Our results show that neural bug finding is surprisingly effective for some bug patterns, sometimes reaching a precision and recall of over 80%, but also that it struggles to understand some program properties obvious to a traditional analysis. A qualitative analysis of the results provides insights into why neural bug finders sometimes work and sometimes do not work. We also identify pitfalls in selecting the code examples used to train and validate neural bug finders, and propose an algorithm for selecting effective training data.
References in corpus (11)
- Natural Language Processing (almost) from Scratch
- Guiding Deep Learning System Testing using Surprise Adequacy
- Automated software vulnerability detection with machine learning
- Generative Code Modeling with Graphs
- Learning to Represent Edits
- Automated Correction for Syntax Errors in Programming Assignments using Recurrent Neural Networks
- Neural Program Repair by Jointly Learning to Localize and Repair
- Context2Name: A Deep Learning-Based Approach to Infer Natural Variable Names from Usage Contexts
- Modeling Vocabulary for Big Code Machine Learning
- Path-Based Function Embedding and its Application to Specification Mining
- End-to-End Prediction of Buffer Overruns from Raw Source Code via Neural Memory Networks