Better Data Labelling with EMBLEM (and how that Impacts Defect Prediction)

Abstract

Standard automatic methods for recognizing problematic development commits can be greatly improved via the incremental application of human+artificial expertise. In this approach, called EMBLEM, an AI tool first explore the software development process to label commits that are most problematic. Humans then apply their expertise to check those labels (perhaps resulting in the AI updating the support vectors within their SVM learner). We recommend this human+AI partnership, for several reasons. When a new domain is encountered, EMBLEM can learn better ways to label which comments refer to real problems. Also, in studies with 9 open source software projects, labelling via EMBLEM's incremental application of human+AI is at least an order of magnitude cheaper than existing methods ($\approx$ eight times). Further, EMBLEM is very effective. For the data sets explored here, EMBLEM better labelling methods significantly improved $P_{opt}20$ and G-scores performance in nearly all the projects studied here.

arXiv comment17 pages, 2 pages references, submitted for TSE journal

References in corpus (10)

• What is Wrong with Topic Modeling? (and How to Fix it Using Search-based Software Engineering)
• Tuning for Software Analytics: is it Really Necessary?
• FAST${}^2$: an Intelligent Assistant for Finding Relevant Papers
• Finding Better Active Learners for Faster Literature Reviews
• Autonomy and Reliability of Continuous Active Learning for Technology-Assisted Review
• How to "DODGE" Complex Software Analytics?
• Improving Vulnerability Inspection Efficiency Using Active Learning
• Why is Differential Evolution Better than Grid Search for Tuning Defect Predictors?
• The Changing Nature of Computational Science Software
• Better Technical Debt Detection via SURVEYing

Cited by in corpus (1)

• The Changing Nature of Computational Science Software