Revisiting Process versus Product Metrics: a Large Scale Analysis

Abstract

Numerous methods can build predictive models from software data. However, what methods and conclusions should we endorse as we move from analytics in-the-small (dealing with a handful of projects) to analytics in-the-large (dealing with hundreds of projects)? To answer this question, we recheck prior small-scale results (about process versus product metrics for defect prediction and the granularity of metrics) using 722,471 commits from 700 Github projects. We find that some analytics in-the-small conclusions still hold when scaling up to analytics in-the-large. For example, like prior work, we see that process metrics are better predictors for defects than product metrics (best process/product-based learners respectively achieve recalls of 98\%/44\% and AUCs of 95\%/54\%, median values). That said, we warn that it is unwise to trust metric importance results from analytics in-the-small studies since those change dramatically when moving to analytics in-the-large. Also, when reasoning in-the-large about hundreds of projects, it is better to use predictions from multiple models (since single model predictions can become confused and exhibit a high variance).

arXiv comment36 pages, 12 figures and 5 tables

References in corpus (8)

• SMOTE: Synthetic Minority Over-sampling Technique
• What is Wrong with Topic Modeling? (and How to Fix it Using Search-based Software Engineering)
• Tuning for Software Analytics: is it Really Necessary?
• Is "Better Data" Better than "Better Data Miners"? (On the Benefits of Tuning SMOTE for Defect Prediction)
• 500+ Times Faster Than Deep Learning (A Case Study Exploring Faster Methods for Text Mining StackOverflow)
• We Don't Need Another Hero? The Impact of "Heroes" on Software Development
• Hyperparameter Optimization for Effort Estimation
• Applications of Psychological Science for Actionable Analytics

Cited by in corpus (1)

• FRUGAL: Unlocking SSL for Software Analytics