InterBERT: Vision-and-Language Interaction for Multi-modal Pretraining

Abstract

Multi-modal pretraining for learning high-level multi-modal representation is a further step towards deep learning and artificial intelligence. In this work, we propose a novel model, namely InterBERT (BERT for Interaction), which is the first model of our series of multimodal pretraining methods M6 (MultiModality-to-MultiModality Multitask Mega-transformer). The model owns strong capability of modeling interaction between the information flows of different modalities. The single-stream interaction module is capable of effectively processing information of multiple modalilties, and the two-stream module on top preserves the independence of each modality to avoid performance downgrade in single-modal tasks. We pretrain the model with three pretraining tasks, including masked segment modeling (MSM), masked region modeling (MRM) and image-text matching (ITM); and finetune the model on a series of vision-and-language downstream tasks. Experimental results demonstrate that InterBERT outperforms a series of strong baselines, including the most recent multi-modal pretraining methods, and the analysis shows that MSM and MRM are effective for pretraining and our method can achieve performances comparable to BERT in single-modal tasks. Besides, we propose a large-scale dataset for multi-modal pretraining in Chinese, and we develop the Chinese InterBERT which is the first Chinese multi-modal pretrained model. We pretrain the Chinese InterBERT on our proposed dataset of 3.1M image-text pairs from the mobile Taobao, the largest Chinese e-commerce platform. We finetune the model for text-based image retrieval, and recently we deployed the model online for topic-based recommendation.

arXiv comment11 pages

References in corpus (12)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Sequence to Sequence Learning with Neural Networks
• Decoupled Weight Decay Regularization
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• Large-Scale Adversarial Training for Vision-and-Language Representation Learning
• ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data
• ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph
• Natural Language Understanding with the Quora Question Pairs Dataset
• Are we pretraining it right? Digging deeper into visio-linguistic pretraining

Cited by in corpus (13)

• VLP: A Survey on Vision-Language Pre-training
• M6: A Chinese Multimodal Pretrainer
• UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning
• MiniVLM: A Smaller and Faster Vision-Language Model
• A Closer Look at the Robustness of Vision-and-Language Pre-trained Models
• Knowledge Perceived Multi-modal Pretraining in E-commerce
• Self-supervised Image-text Pre-training With Mixed Data In Chest X-rays
• Kaleido-BERT: Vision-Language Pre-training on Fashion Domain
• Inferring spatial relations from textual descriptions of images
• Deep Insights into Cognitive Decline: A Survey of Leveraging Non-Intrusive Modalities with Deep Learning Techniques
• Achieving Human Parity on Visual Question Answering
• Learning Relation Alignment for Calibrated Cross-modal Retrieval
• GiBERT: Introducing Linguistic Knowledge into BERT through a Lightweight Gated Injection Method