The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes
arXiv:2005.04790
Abstract
This work proposes a new challenge set for multimodal classification, focusing on detecting hate speech in multimodal memes. It is constructed such that unimodal models struggle and only multimodal models can succeed: difficult examples ("benign confounders") are added to the dataset to make it hard to rely on unimodal signals. The task requires subtle reasoning, yet is straightforward to evaluate as a binary classification problem. We provide baseline performance numbers for unimodal models, as well as for multimodal models with various degrees of sophistication. We find that state-of-the-art methods perform poorly compared to humans (64.73% vs. 84.7% accuracy), illustrating the difficulty of the task and highlighting the challenge that this important problem poses to the community.
arXiv commentNeurIPS 2020
References in corpus (26)
- Adam: A Method for Stochastic Optimization
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Decoupled Weight Decay Regularization
- Microsoft COCO Captions: Data Collection and Evaluation Server
- YFCC100M: The New Data in Multimedia Research
- VisualBERT: A Simple and Performant Baseline for Vision and Language
- Fake News Detection on Social Media: A Data Mining Perspective
- Aggregated Residual Transformations for Deep Neural Networks
- Simple Baseline for Visual Question Answering
- Learning the Difference that Makes a Difference with Counterfactually-Augmented Data
- Supervised Multimodal Bitransformers for Classifying Images and Text
- Exploring Nearest Neighbor Approaches for Image Captioning
- Visual Entailment: A Novel Task for Fine-Grained Image Understanding
- Visual Storytelling
- Prediction of Cyberbullying Incidents on the Instagram Social Network
- Hate Speech in Pixels: Detection of Offensive Memes towards Automatic Moderation
- Characterizing and Detecting Hateful Users on Twitter
- Gated Multimodal Units for Information Fusion
- Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
- Equal But Not The Same: Understanding the Implicit Relationship Between Persuasive Images and Text
- An empirical study on the effectiveness of images in Multimodal Neural Machine Translation
- Detecting Hate Speech in Social Media
- Are we pretraining it right? Digging deeper into visio-linguistic pretraining
- Virtual Embodiment: A Scalable Long-Term Strategy for Artificial Intelligence Research
- Interpretable Multi-Modal Hate Speech Detection
- On the adequacy of untuned warmup for adaptive optimization
Cited by in corpus (36)
- Learning Transferable Visual Models From Natural Language Supervision
- Directions in Abusive Language Training Data: Garbage In, Garbage Out
- Disentangling Hate in Online Memes
- Confronting Abusive Language Online: A Survey from the Ethical and Human Rights Perspective
- Tackling Online Abuse: A Survey of Automated Abuse Detection Methods
- Detecting Hate Speech in Memes Using Multimodal Deep Learning Approaches: Prize-winning solution to Hateful Memes Challenge
- A Multimodal Framework for the Detection of Hateful Memes
- Enhance Multimodal Transformer With External Label And In-Domain Pretrain: Hateful Meme Challenge Winning Solution
- Vilio: State-of-the-art Visio-Linguistic Models applied to Hateful Memes
- Detecting Hate Speech in Multi-modal Memes
- Measuring a hate speech spectrum with faceted Rasch item response theory and perspective-aware, explainable-by-design deep learning
- A Multimodal Memes Classification: A Survey and Open Research Issues
- Benchmark dataset of memes with text transcriptions for automatic detection of multi-modal misogynistic content
- HateCheck: Functional Tests for Hate Speech Detection Models
- Detecting Hateful Memes Using a Multimodal Deep Ensemble
- PromptMTopic: Unsupervised Multimodal Topic Modeling of Memes using Large Language Models
- Hateful Memes Detection via Complementary Visual and Linguistic Networks
- Classification of Multimodal Hate Speech -- The Winning Solution of Hateful Memes Challenge
- A Survey on Multimodal Disinformation Detection
- Sentiment Analysis of Fashion Related Posts in Social Media
- Multimodal Fusion Refiner Networks
- Dissecting the Meme Magic: Understanding Indicators of Virality in Image Memes
- TotalDefMeme: A Multi-Attribute Meme dataset on Total Defence in Singapore
- Cross-Modal Transfer from Memes to Videos: Addressing Data Scarcity in Hateful Video Detection
- Unsupervised Vision-and-Language Pre-training Without Parallel Images and Captions
- Multimodal Learning for Hateful Memes Detection
- MATK: The Meme Analytical Tool Kit
- Towards A Multi-agent System for Online Hate Speech Detection
- Enhance Multimodal Model Performance with Data Augmentation: Facebook Hateful Meme Challenge Solution
- Caption Enriched Samples for Improving Hateful Memes Detection
- Adversarial Evaluation of Multimodal Models under Realistic Gray Box Assumption
- Overview of the HASOC track at FIRE 2020: Hate Speech and Offensive Content Identification in Indo-European Languages
- MOMENTA: A Multimodal Framework for Detecting Harmful Memes and Their Targets
- Dendritic Self-Organizing Maps for Continual Learning
- Human-in-the-Loop Disinformation Detection: Stance, Sentiment, or Something Else?
- AOMD: An Analogy-aware Approach to Offensive Meme Detection on Social Media