Human-in-the-Loop for Data Collection: a Multi-Target Counter Narrative Dataset to Fight Online Hate Speech

Abstract

Undermining the impact of hateful content with informed and non-aggressive responses, called counter narratives, has emerged as a possible solution for having healthier online communities. Thus, some NLP studies have started addressing the task of counter narrative generation. Although such studies have made an effort to build hate speech / counter narrative (HS/CN) datasets for neural generation, they fall short in reaching either high-quality and/or high-quantity. In this paper, we propose a novel human-in-the-loop data collection methodology in which a generative language model is refined iteratively by using its own data from the previous loops to generate new training samples that experts review and/or post-edit. Our experiments comprised several loops including dynamic variations. Results show that the methodology is scalable and facilitates diverse, novel, and cost-effective data collection. To our knowledge, the resulting dataset is the only expert-based multi-target HS/CN dataset available to the community.

To appear at ACL 2021 (long paper)

References in corpus (5)

• The Curious Case of Neural Text Degeneration
• Directions in Abusive Language Training Data: Garbage In, Garbage Out
• Release Strategies and the Social Impacts of Language Models
• Measuring the Reliability of Hate Speech Annotations: The Case of the European Refugee Crisis
• Analyzing the hate and counter speech accounts on Twitter

Cited by in corpus (5)

• On the rise of fear speech in online social media
• SOLD: Sinhala Offensive Language Dataset
• SynthBio: A Case Study in Human-AI Collaborative Curation of Text Datasets
• Perceiving and Countering Hate: The Role of Identity in Online Responses
• LEAP: LLM-powered End-to-end Automatic Library for Processing Social Science Queries on Unstructured Data