Character-level HyperNetworks for Hate Speech Detection

Abstract

The massive spread of hate speech, hateful content targeted at specific subpopulations, is a problem of critical social importance. Automated methods of hate speech detection typically employ state-of-the-art deep learning (DL)-based text classifiers-large pretrained neural language models of over 100 million parameters, adapting these models to the task of hate speech detection using relevant labeled datasets. Unfortunately, there are only a few public labeled datasets of limited size that are available for this purpose. We make several contributions with high potential for advancing this state of affairs. We present HyperNetworks for hate speech detection, a special class of DL networks whose weights are regulated by a small-scale auxiliary network. These architectures operate at character-level, as opposed to word or subword-level, and are several orders of magnitude smaller compared to the popular DL classifiers. We further show that training hate detection classifiers using additional large amounts of automatically generated examples is beneficial in general, yet this practice especially boosts the performance of the proposed HyperNetworks. We report the results of extensive experiments, assessing the performance of multiple neural architectures on hate detection using five public datasets. The assessed methods include the pretrained language models of BERT, RoBERTa, ALBERT, MobileBERT and CharBERT, a variant of BERT that incorporates character alongside subword embeddings. In addition to the traditional setup of within-dataset evaluation, we perform cross-dataset evaluation experiments, testing the generalization of the various models in conditions of data shift. Our results show that the proposed HyperNetworks achieve performance that is competitive, and better in some cases, than these pretrained language models, while being smaller by orders of magnitude.

References in corpus (8)

• Distributed Representations of Words and Phrases and their Compositionality
• Character-level Convolutional Networks for Text Classification
• Mobile Edge Computing: A Survey on Architecture and Computation Offloading
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• Deep Learning for Hate Speech Detection in Tweets
• Hate Speech Detection and Racial Bias Mitigation in Social Media based on BERT model
• CharBERT: Character-aware Pre-trained Language Model
• As Good as New. How to Successfully Recycle English GPT-2 to Make Models for Other Languages

Cited by in corpus (2)

• A Brief Review of Hypernetworks in Deep Learning
• A survey of textual cyber abuse detection using cutting-edge language models and large language models