On the Replicability and Reproducibility of Deep Learning in Software Engineering
arXiv:2006.14244 · doi:10.1145/3477535
Abstract
Deep learning (DL) techniques have gained significant popularity among software engineering (SE) researchers in recent years. This is because they can often solve many SE challenges without enormous manual feature engineering effort and complex domain knowledge. Although many DL studies have reported substantial advantages over other state-of-the-art models on effectiveness, they often ignore two factors: (1) replicability - whether the reported experimental result can be approximately reproduced in high probability with the same DL model and the same data; and (2) reproducibility - whether one reported experimental findings can be reproduced by new experiments with the same experimental protocol and DL model, but different sampled real-world data. Unlike traditional machine learning (ML) models, DL studies commonly overlook these two factors and declare them as minor threats or leave them for future work. This is mainly due to high model complexity with many manually set parameters and the time-consuming optimization process. In this study, we conducted a literature review on 93 DL studies recently published in twenty SE journals or conferences. Our statistics show the urgency of investigating these two factors in SE. Moreover, we re-ran four representative DL models in SE. Experimental results show the importance of replicability and reproducibility, where the reported performance of a DL model could not be replicated for an unstable optimization process. Reproducibility could be substantially compromised if the model training is not convergent, or if performance is sensitive to the size of vocabulary and testing data. It is therefore urgent for the SE community to provide a long-lasting link to a replication package, enhance DL-based solution stability and convergence, and avoid performance sensitivity on different sampled data.
References in corpus (9)
- Deep Learning in Neural Networks: An Overview
- Software Engineering Challenges of Deep Learning
- Learn&Fuzz: Machine Learning for Input Fuzzing
- PatchNet: Hierarchical Deep Learning-Based Stable Patch Identification for the Linux Kernel
- Open Vocabulary Learning on Source Code with a Graph-Structured Cache
- Neural Guided Constraint Logic Programming for Program Synthesis
- Neural-Augmented Static Analysis of Android Communication
- A Neural Model for Generating Natural Language Summaries of Program Subroutines
- On Learning Meaningful Code Changes via Neural Machine Translation