ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data

Abstract

In this paper, we introduce a new vision-language pre-trained model -- ImageBERT -- for image-text joint embedding. Our model is a Transformer-based model, which takes different modalities as input and models the relationship between them. The model is pre-trained on four tasks simultaneously: Masked Language Modeling (MLM), Masked Object Classification (MOC), Masked Region Feature Regression (MRFR), and Image Text Matching (ITM). To further enhance the pre-training quality, we have collected a Large-scale weAk-supervised Image-Text (LAIT) dataset from Web. We first pre-train the model on this dataset, then conduct a second stage pre-training on Conceptual Captions and SBU Captions. Our experiments show that multi-stage pre-training strategy outperforms single-stage pre-training. We also fine-tune and evaluate our pre-trained ImageBERT model on image retrieval and text retrieval tasks, and achieve new state-of-the-art results on both MSCOCO and Flickr30k datasets.

References in corpus (10)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• Microsoft COCO Captions: Data Collection and Evaluation Server
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• VQA: Visual Question Answering
• VL-BERT: Pre-training of Generic Visual-Linguistic Representations
• DenseNet: Implementing Efficient ConvNet Descriptor Pyramids
• Deep Visual-Semantic Alignments for Generating Image Descriptions
• Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training
• Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering

Cited by in corpus (29)

• Learning Transferable Visual Models From Natural Language Supervision
• Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
• ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
• Florence: A New Foundation Model for Computer Vision
• Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers
• VLP: A Survey on Vision-Language Pre-training
• Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training
• FILIP: Fine-grained Interactive Language-Image Pre-Training
• WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning
• WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research
• WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Training
• GPT-4 Enhanced Multimodal Grounding for Autonomous Driving: Leveraging Cross-Modal Attention with Large Language Models
• InterBERT: Vision-and-Language Interaction for Multi-modal Pretraining
• A Multimodal Memes Classification: A Survey and Open Research Issues
• The Chess Transformer: Mastering Play using Generative Language Models
• Pre-Trained Models: Past, Present and Future
• Kaleido-BERT: Vision-Language Pre-training on Fashion Domain
• Effective End-to-End Vision Language Pretraining with Semantic Visual Loss
• lamBERT: Language and Action Learning Using Multimodal BERT
• Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks
• SHAMSUL: Systematic Holistic Analysis to investigate Medical Significance Utilizing Local interpretability methods in deep learning for chest radiography pathology prediction
• Relation Transformer Network
• WikiMuTe: A web-sourced dataset of semantic descriptions for music audio
• Video Transformer for Deepfake Detection with Incremental Learning
• LAMP: Label Augmented Multimodal Pretraining
• Predicting times of waiting on red signals using BERT
• Exceeding the Limits of Visual-Linguistic Multi-Task Learning
• Survey of Visual-Semantic Embedding Methods for Zero-Shot Image Retrieval
• Multi-stage Pre-training over Simplified Multimodal Pre-training Models