A Multimodal Memes Classification: A Survey and Open Research Issues

Abstract

Memes are graphics and text overlapped so that together they present concepts that become dubious if one of them is absent. It is spread mostly on social media platforms, in the form of jokes, sarcasm, motivating, etc. After the success of BERT in Natural Language Processing (NLP), researchers inclined to Visual-Linguistic (VL) multimodal problems like memes classification, image captioning, Visual Question Answering (VQA), and many more. Unfortunately, many memes get uploaded each day on social media platforms that need automatic censoring to curb misinformation and hate. Recently, this issue has attracted the attention of researchers and practitioners. State-of-the-art methods that performed significantly on other VL dataset, tends to fail on memes classification. In this context, this work aims to conduct a comprehensive study on memes classification, generally on the VL multimodal problems and cutting edge solutions. We propose a generalized framework for VL problems. We cover the early and next-generation works on VL problems. Finally, we identify and articulate several open research issues and challenges. This is the first study that presents the generalized view of the advanced classification techniques concerning memes classification to the best of our knowledge. We believe this study presents a clear road-map for the Machine Learning (ML) research community to implement and enhance memes classification techniques.

This is a survey paper on recent state of the art VL models that can be used for memes classification. it has 15 pages and 2 figures

References in corpus (9)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Large-Scale Adversarial Training for Vision-and-Language Representation Learning
• Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers
• Exploring Nearest Neighbor Approaches for Image Captioning
• ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data
• The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes
• HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training
• VD-BERT: A Unified Vision and Dialog Transformer with BERT

Cited by in corpus (2)

• Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training
• Literature review on vulnerability detection using NLP technology