Multimodal Learning for Hateful Memes Detection
arXiv:2011.12870
Abstract
Memes are used for spreading ideas through social networks. Although most memes are created for humor, some memes become hateful under the combination of pictures and text. Automatically detecting the hateful memes can help reduce their harmful social impact. Unlike the conventional multimodal tasks, where the visual and textual information is semantically aligned, the challenge of hateful memes detection lies in its unique multimodal information. The image and text in memes are weakly aligned or even irrelevant, which requires the model to understand the content and perform reasoning over multiple modalities. In this paper, we focus on multimodal hateful memes detection and propose a novel method that incorporates the image captioning process into the memes detection process. We conduct extensive experiments on multimodal meme datasets and illustrated the effectiveness of our approach. Our model achieves promising results on the Hateful Memes Detection Challenge.
References in corpus (14)
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Microsoft COCO Captions: Data Collection and Evaluation Server
- VisualBERT: A Simple and Performant Baseline for Vision and Language
- VQA: Visual Question Answering
- Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answering
- Exploring Models and Data for Image Question Answering
- Hadamard Product for Low-rank Bilinear Pooling
- Deep Visual-Semantic Alignments for Generating Image Descriptions
- The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes
- Prediction of Cyberbullying Incidents on the Instagram Social Network
- Characterizing and Detecting Hateful Users on Twitter
- Detecting Hate Speech in Social Media