Assessing Software Defection Prediction Performance: Why Using the Matthews Correlation Coefficient Matters

Abstract

Context: There is considerable diversity in the range and design of computational experiments to assess classifiers for software defect prediction. This is particularly so, regarding the choice of classifier performance metrics. Unfortunately some widely used metrics are known to be biased, in particular F1. Objective: We want to understand the extent to which the widespread use of the F1 renders empirical results in software defect prediction unreliable. Method: We searched for defect prediction studies that report both F1 and the Matthews correlation coefficient (MCC). This enabled us to determine the proportion of results that are consistent between both metrics and the proportion that change. Results: Our systematic review identifies 8 studies comprising 4017 pairwise results. Of these results, the direction of the comparison changes in 23% of the cases when the unbiased MCC metric is employed. Conclusion: We find compelling reasons why the choice of classification performance metric matters, specifically the biased and misleading F1 metric should be deprecated.

EASE 2020, April 15--17, 2020, Trondheim, Norway

References in corpus (1)

• Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation

Cited by in corpus (5)

• Area under the ROC Curve has the Most Consistent Evaluation for Binary Classification
• On the Cross-Dataset Generalization of Machine Learning for Network Intrusion Detection
• Problems with SZZ and Features: An empirical study of the state of practice of defect prediction data collection
• Test case prioritization using test case diversification and fault-proneness estimations
• Feature Sets in Just-in-Time Defect Prediction: An Empirical Evaluation