Supervised Multimodal Bitransformers for Classifying Images and Text

Abstract

Self-supervised bidirectional transformer models such as BERT have led to dramatic improvements in a wide variety of textual classification tasks. The modern digital world is increasingly multimodal, however, and textual information is often accompanied by other modalities such as images. We introduce a supervised multimodal bitransformer model that fuses information from text and image encoders, and obtain state-of-the-art performance on various multimodal classification benchmark tasks, outperforming strong baselines, including on hard test sets specifically designed to measure multimodal performance.

arXiv commentRejected from EMNLP, twice

References in corpus (13)

• Adam: A Method for Stochastic Optimization
• Distributed Representations of Words and Phrases and their Compositionality
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• VQA: Visual Question Answering
• Zero-Shot Learning Through Cross-Modal Transfer
• Skip-Thought Vectors
• VSE++: Improving Visual-Semantic Embeddings with Hard Negatives
• Semi-supervised Sequence Learning
• A large annotated corpus for learning natural language inference
• Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks
• LXMERT: Learning Cross-Modality Encoder Representations from Transformers
• VideoBERT: A Joint Model for Video and Language Representation Learning
• What Makes Training Multi-Modal Classification Networks Hard?

Cited by in corpus (27)

• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes
• Pretrained Transformers as Universal Computation Engines
• Trusted Multi-View Classification
• A Multimodal Framework for the Detection of Hateful Memes
• Are we pretraining it right? Digging deeper into visio-linguistic pretraining
• MultiBench: Multiscale Benchmarks for Multimodal Representation Learning
• Human-Adversarial Visual Question Answering
• Detection of Illicit Drug Trafficking Events on Instagram: A Deep Multimodal Multilabel Learning Approach
• Improving Generalization for Multimodal Fake News Detection
• Improving Multimodal Accuracy Through Modality Pre-training and Attention
• Moderating New Waves of Online Hate with Chain-of-Thought Reasoning in Large Language Models
• A Survey on Multimodal Disinformation Detection
• lamBERT: Language and Action Learning Using Multimodal BERT
• Multimodal Fusion Refiner Networks
• Multimodal Conditionality for Natural Language Generation
• DM${}^2$S${}^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
• TrollsWithOpinion: A Dataset for Predicting Domain-specific Opinion Manipulation in Troll Memes
• Enhance Multimodal Model Performance with Data Augmentation: Facebook Hateful Meme Challenge Solution
• Enhancing Cross-Modal Contextual Congruence for Crowdfunding Success using Knowledge-infused Learning
• Exceeding the Limits of Visual-Linguistic Multi-Task Learning
• Understanding of Emotion Perception from Art
• Deep Multi-Modal Sets
• Cross-Modal Retrieval Augmentation for Multi-Modal Classification
• Detecting Propaganda Techniques in Memes
• DRDF: Determining the Importance of Different Multimodal Information with Dual-Router Dynamic Framework
• Adversarial Evaluation of Multimodal Models under Realistic Gray Box Assumption