Can We Use SE-specific Sentiment Analysis Tools in a Cross-Platform Setting?

Abstract

In this paper, we address the problem of using sentiment analysis tools 'off-the-shelf,' that is when a gold standard is not available for retraining. We evaluate the performance of four SE-specific tools in a cross-platform setting, i.e., on a test set collected from data sources different from the one used for training. We find that (i) the lexicon-based tools outperform the supervised approaches retrained in a cross-platform setting and (ii) retraining can be beneficial in within-platform settings in the presence of robust gold standard datasets, even using a minimal training set. Based on our empirical findings, we derive guidelines for reliable use of sentiment analysis tools in software engineering.

12 pages

References in corpus (6)

• SMOTE: Synthetic Minority Over-sampling Technique
• Machine Learning in Automated Text Categorization
• Sentiment Polarity Detection for Software Development
• Is "Better Data" Better than "Better Data Miners"? (On the Benefits of Tuning SMOTE for Defect Prediction)
• How to Ask for Technical Help? Evidence-based Guidelines for Writing Questions on Stack Overflow
• Mining Valence, Arousal, and Dominance - Possibilities for Detecting Burnout and Productivity?

Cited by in corpus (12)

• The "Shut the f**k up" Phenomenon: Characterizing Incivility in Open Source Code Review Discussions
• Revisiting Sentiment Analysis for Software Engineering in the Era of Large Language Models
• Development and Application of Sentiment Analysis Tools in Software Engineering: A Systematic Literature Review
• Assessment of Off-the-Shelf SE-specific Sentiment Analysis Tools: An Extended Replication Study
• Cross-Dataset Design Discussion Mining
• Are Prompts All You Need? Evaluating Prompt-Based Large Language Models (LLM)s for Software Requirements Classification
• Assessing the Use of AutoML for Data-Driven Software Engineering
• On the Limitations of Combining Sentiment Analysis Tools in a Cross-Platform Setting
• Predicting Community Smells' Occurrence on Individual Developers by Sentiments
• What Characteristics Make ChatGPT Effective for Software Issue Resolution? An Empirical Study of Task, Project, and Conversational Signals in GitHub Issues
• Sentiment analysis for software engineering: How far can zero-shot learning (ZSL) go?
• Exploiting the Unique Expression for Improved Sentiment Analysis in Software Engineering Text