WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Training

Abstract

Multi-modal pre-training models have been intensively explored to bridge vision and language in recent years. However, most of them explicitly model the cross-modal interaction between image-text pairs, by assuming that there exists strong semantic correlation between the text and image modalities. Since this strong assumption is often invalid in real-world scenarios, we choose to implicitly model the cross-modal correlation for large-scale multi-modal pre-training, which is the focus of the Chinese project `WenLan' led by our team. Specifically, with the weak correlation assumption over image-text pairs, we propose a two-tower pre-training model called BriVL within the cross-modal contrastive learning framework. Unlike OpenAI CLIP that adopts a simple contrastive learning method, we devise a more advanced algorithm by adapting the latest method MoCo into the cross-modal scenario. By building a large queue-based dictionary, our BriVL can incorporate more negative samples in limited GPU resources. We further construct a large Chinese multi-source image-text dataset called RUC-CAS-WenLan for pre-training our BriVL model. Extensive experiments demonstrate that the pre-trained BriVL model outperforms both UNITER and OpenAI CLIP on various downstream tasks.

arXiv commentThis paper is the outcome of the Chinese multi-modal pre-training project called 'WenLan'

References in corpus (12)

• Learning Transferable Visual Models From Natural Language Supervision
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Language Models are Few-Shot Learners
• Generative Adversarial Text to Image Synthesis
• REALM: Retrieval-Augmented Language Model Pre-Training
• Exploring Simple Siamese Representation Learning
• Large-Scale Adversarial Training for Vision-and-Language Representation Learning
• ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data
• AI Challenger : A Large-scale Dataset for Going Deeper in Image Understanding
• ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph
• X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers
• XGPT: Cross-modal Generative Pre-Training for Image Captioning

Cited by in corpus (11)

• Information Retrieval: Recent Advances and Beyond
• CenterCLIP: Token Clustering for Efficient Text-Video Retrieval
• Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm
• COCOA: Cross Modality Contrastive Learning for Sensor Data
• Text-Based Person Search with Limited Data
• Scaling Up Vision-Language Pre-training for Image Captioning
• Pre-Trained Models: Past, Present and Future
• Multi-modal Self-supervised Pre-training for Regulatory Genome Across Cell Types
• INTERN: A New Learning Paradigm Towards General Vision
• EfficientCLIP: Efficient Cross-Modal Pre-training by Ensemble Confident Learning and Language Modeling
• Text2Poster: Laying out Stylized Texts on Retrieved Images