Multilingual Twitter Corpus and Baselines for Evaluating Demographic Bias in Hate Speech Recognition

Abstract

Existing research on fairness evaluation of document classification models mainly uses synthetic monolingual data without ground truth for author demographic attributes. In this work, we assemble and publish a multilingual Twitter corpus for the task of hate speech detection with inferred four author demographic factors: age, country, gender and race/ethnicity. The corpus covers five languages: English, Italian, Polish, Portuguese and Spanish. We evaluate the inferred demographic labels with a crowdsourcing platform, Figure Eight. To examine factors that can cause biases, we take an empirical analysis of demographic predictability on the English corpus. We measure the performance of four popular document classifiers and evaluate the fairness and bias of the baseline classifiers on the author-level demographic attributes.

arXiv commentAccepted at LREC 2020

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• Scikit-learn: Machine Learning in Python
• Distributed Representations of Words and Phrases and their Compositionality
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• NLTK: The Natural Language Toolkit
• Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings
• Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypes
• The Frontiers of Fairness in Machine Learning
• How to Fine-Tune BERT for Text Classification?

Cited by in corpus (4)

• Confronting Abusive Language Online: A Survey from the Ethical and Human Rights Perspective
• Deep Learning Models for Multilingual Hate Speech Detection
• Language-Driven Image Style Transfer
• The Authors Matter: Understanding and Mitigating Implicit Bias in Deep Text Classification