Incivility Detection in Open Source Code Review and Issue Discussions
arXiv:2206.13429 · doi:10.1016/j.jss.2023.111935
Abstract
Given the democratic nature of open source development, code review and issue discussions may be uncivil. Incivility, defined as features of discussion that convey an unnecessarily disrespectful tone, can have negative consequences to open source communities. To prevent or minimize these negative consequences, open source platforms have included mechanisms for removing uncivil language from the discussions. However, such approaches require manual inspection, which can be overwhelming given the large number of discussions. To help open source communities deal with this problem, in this paper, we aim to compare six classical machine learning models with BERT to detect incivility in open source code review and issue discussions. Furthermore, we assess if adding contextual information improves the models' performance and how well the models perform in a cross-platform setting. We found that BERT performs better than classical machine learning models, with a best F1-score of 0.95. Furthermore, classical machine learning models tend to underperform to detect non-technical and civil discussions. Our results show that adding the contextual information to BERT did not improve its performance and that none of the analyzed classifiers had an outstanding performance in a cross-platform setting. Finally, we provide insights into the tones that the classifiers misclassify.
arXiv comment18 pages
References in corpus (7)
- SMOTE: Synthetic Minority Over-sampling Technique
- NLTK: The Natural Language Toolkit
- Well-Read Students Learn Better: On the Importance of Pre-training Compact Models
- Source Code Properties of Defective Infrastructure as Code Scripts
- The "Shut the f**k up" Phenomenon: Characterizing Incivility in Open Source Code Review Discussions
- Opinion Conflicts: An Effective Route to Detect Incivility in Twitter
- How heated is it? Understanding GitHub locked issues