Disentangling Hate in Online Memes

Abstract

Hateful and offensive content detection has been extensively explored in a single modality such as text. However, such toxic information could also be communicated via multimodal content such as online memes. Therefore, detecting multimodal hateful content has recently garnered much attention in academic and industry research communities. This paper aims to contribute to this emerging research topic by proposing DisMultiHate, which is a novel framework that performed the classification of multimodal hateful content. Specifically, DisMultiHate is designed to disentangle target entities in multimodal memes to improve hateful content classification and explainability. We conduct extensive experiments on two publicly available hateful and offensive memes datasets. Our experiment results show that DisMultiHate is able to outperform state-of-the-art unimodal and multimodal baselines in the hateful meme classification task. Empirical case studies were also conducted to demonstrate DisMultiHate's ability to disentangle target entities in memes and ultimately showcase DisMultiHate's explainability of the multimodal hateful content classification task.

arXiv commentPaper accepted in ACM Multimedia 2021

References in corpus (27)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Categorical Reparameterization with Gumbel-Softmax
• ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• Deep Learning for Hate Speech Detection in Tweets
• VL-BERT: Pre-training of Generic Visual-Linguistic Representations
• Large-Scale Adversarial Training for Vision-and-Language Representation Learning
• Understanding disentangling in $\beta$-VAE
• The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes
• FairFace: Face Attribute Dataset for Balanced Race, Gender, and Age
• Multi-Level Variational Autoencoder: Learning Disentangled Representations from Grouped Observations
• ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph
• Learning Disentangled Representations for Recommendation
• Mean Birds: Detecting Aggression and Bullying on Twitter
• All You Need is "Love": Evading Hate-speech Detection
• Learning Disentangled Joint Continuous and Discrete Representations
• Bayesian representation learning with oracle constraints
• Detecting Hate Speech in Memes Using Multimodal Deep Learning Approaches: Prize-winning solution to Hateful Memes Challenge
• A Multimodal Framework for the Detection of Hateful Memes
• Enhance Multimodal Transformer With External Label And In-Domain Pretrain: Hateful Meme Challenge Winning Solution
• Detecting Hate Speech in Multi-modal Memes
• Vilio: State-of-the-art Visio-Linguistic Models applied to Hateful Memes
• Learning to Linearize Under Uncertainty
• Detecting Hateful Memes Using a Multimodal Deep Ensemble
• Hateful Memes Detection via Complementary Visual and Linguistic Networks
• Classification of Multimodal Hate Speech -- The Winning Solution of Hateful Memes Challenge
• Multimodal Learning for Hateful Memes Detection

Cited by in corpus (4)

• PromptMTopic: Unsupervised Multimodal Topic Modeling of Memes using Large Language Models
• Toxic Memes: A Survey of Computational Perspectives on the Detection and Explanation of Meme Toxicities
• TotalDefMeme: A Multi-Attribute Meme dataset on Total Defence in Singapore
• MATK: The Meme Analytical Tool Kit