Caption Enriched Samples for Improving Hateful Memes Detection
arXiv:2109.10649
Abstract
The recently introduced hateful meme challenge demonstrates the difficulty of determining whether a meme is hateful or not. Specifically, both unimodal language models and multimodal vision-language models cannot reach the human level of performance. Motivated by the need to model the contrast between the image content and the overlayed text, we suggest applying an off-the-shelf image captioning tool in order to capture the first. We demonstrate that the incorporation of such automatic captions during fine-tuning improves the results for various unimodal and multimodal models. Moreover, in the unimodal case, continuing the pre-training of language models on augmented and original caption pairs, is highly beneficial to the classification accuracy.
arXiv commentEMNLP 2021
References in corpus (5)
- Microsoft COCO Captions: Data Collection and Evaluation Server
- VisualBERT: A Simple and Performant Baseline for Vision and Language
- Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge
- The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes
- Vilio: State-of-the-art Visio-Linguistic Models applied to Hateful Memes