MOMENTA: A Multimodal Framework for Detecting Harmful Memes and Their Targets

Abstract

Internet memes have become powerful means to transmit political, psychological, and socio-cultural ideas. Although memes are typically humorous, recent days have witnessed an escalation of harmful memes used for trolling, cyberbullying, and abuse. Detecting such memes is challenging as they can be highly satirical and cryptic. Moreover, while previous work has focused on specific aspects of memes such as hate speech and propaganda, there has been little work on harm in general. Here, we aim to bridge this gap. We focus on two tasks: (i)detecting harmful memes, and (ii)identifying the social entities they target. We further extend a recently released HarMeme dataset, which covered COVID-19, with additional memes and a new topic: US politics. To solve these tasks, we propose MOMENTA (MultimOdal framework for detecting harmful MemEs aNd Their tArgets), a novel multimodal deep neural network that uses global and local perspectives to detect harmful memes. MOMENTA systematically analyzes the local and the global perspective of the input meme (in both modalities) and relates it to the background context. MOMENTA is interpretable and generalizable, and our experiments show that it outperforms several strong rivaling approaches.

arXiv commentThe paper has been accepted in the Findings of Empirical Methods in Natural Language Processing (EMNLP), 2021

References in corpus (11)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Learning Transferable Visual Models From Natural Language Supervision
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• Hate Speech Detection and Racial Bias Mitigation in Social Media based on BERT model
• The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes
• Detecting Hate Speech in Memes Using Multimodal Deep Learning Approaches: Prize-winning solution to Hateful Memes Challenge
• A Multimodal Framework for the Detection of Hateful Memes
• Vilio: State-of-the-art Visio-Linguistic Models applied to Hateful Memes
• Detecting Hateful Memes Using a Multimodal Deep Ensemble
• Exercise? I thought you said 'Extra Fries': Leveraging Sentence Demarcations and Multi-hop Attention for Meme Affect Analysis