Multilingual Twitter Sentiment Classification: The Role of Human Annotators

Abstract

What are the limits of automated Twitter sentiment classification? We analyze a large set of manually labeled tweets in different languages, use them as training data, and construct automated classification models. It turns out that the quality of classification models depends much more on the quality and size of training data than on the type of the model trained. Experimental results indicate that there is no statistically significant difference between the performance of the top classification models. We quantify the quality of training data by applying various annotator agreement measures, and identify the weakest points of different datasets. We show that the model performance approaches the inter-annotator agreement when the size of the training set is sufficiently large. However, it is crucial to regularly monitor the self- and inter-annotator agreements since this improves the training datasets and consequently the model performance. Finally, we show that there is strong evidence that humans perceive the sentiment classes (negative, neutral, and positive) as ordered.

References in corpus (4)

• Sentiment of Emojis
• The Effects of Twitter Sentiment on Stock Price Returns
• Emotional Dynamics in the Age of Misinformation
• Twitter Sentiment Analysis: Lexicon Method, Machine Learning Method and Their Combination

Cited by in corpus (19)

• Creating emoji lexica from unsupervised sentiment analysis of their descriptions
• "Garbage In, Garbage Out" Revisited: What Do Machine Learning Application Papers Report About Human-Labeled Training Data?
• Cohesion and Coalition Formation in the European Parliament: Roll-Call Votes and Twitter Activities
• Sentiment Analysis of Code-Mixed Languages leveraging Resource Rich Languages
• Twitter Sentiment around the Earnings Announcement Events
• Retweet communities reveal the main sources of hate speech
• How to evaluate sentiment classifiers for Twitter time-ordered data?
• Cross-lingual alignments of ELMo contextual embeddings
• EvoMSA: A Multilingual Evolutionary Approach for Sentiment Analysis
• Online Hate: Behavioural Dynamics and Relationship with Misinformation
• Emotions are Universal: Learning Sentiment Based Representations of Resource-Poor Languages using Siamese Networks
• XED: A Multilingual Dataset for Sentiment Analysis and Emotion Detection
• A New Statistical Approach for Comparing Algorithms for Lexicon Based Sentiment Analysis
• Integrating Crowdsourcing and Active Learning for Classification of Work-Life Events from Tweets
• Multilingual, Temporal and Sentimental Distant-Reading of City Events
• Skin Tone Emoji and Sentiment on Twitter
• Forex trading and Twitter: Spam, bots, and reputation manipulation
• Garbage In, Garbage Out? Do Machine Learning Application Papers in Social Computing Report Where Human-Labeled Training Data Comes From?
• Cross-lingual Transfer for Text Classification with Dictionary-based Heterogeneous Graph