M3P: Learning Universal Representations via Multitask Multilingual Multimodal Pre-training

Abstract

We present M3P, a Multitask Multilingual Multimodal Pre-trained model that combines multilingual pre-training and multimodal pre-training into a unified framework via multitask pre-training. Our goal is to learn universal representations that can map objects occurred in different modalities or texts expressed in different languages into a common semantic space. In addition, to explicitly encourage fine-grained alignment between images and non-English languages, we also propose Multimodal Code-switched Training (MCT) to combine monolingual pre-training and multimodal pre-training via a code-switch strategy. Experiments are performed on the multilingual image retrieval task across two benchmark datasets, including MSCOCO and Multi30K. M3P can achieve comparable results for English and new state-of-the-art results for non-English languages.

arXiv commentAccepted to CVPR 2021

References in corpus (12)

• Microsoft COCO Captions: Data Collection and Evaluation Server
• Large-Scale Adversarial Training for Vision-and-Language Representation Learning
• UNITER: UNiversal Image-TExt Representation Learning
• XNLI: Evaluating Cross-lingual Sentence Representations
• ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph
• Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training
• Imagination improves Multimodal Translation
• Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
• XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation
• STAIR Captions: Constructing a Large-Scale Japanese Image Caption Dataset
• XGPT: Cross-modal Generative Pre-Training for Image Captioning
• CoSDA-ML: Multi-Lingual Code-Switching Data Augmentation for Zero-Shot Cross-Lingual NLP

Cited by in corpus (8)

• Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
• Cross-Lingual Cross-Modal Retrieval with Noise-Robust Learning
• IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages
• Pre-Trained Models: Past, Present and Future
• MURAL: Multimodal, Multitask Retrieval Across Languages
• UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training
• Rethinking the constraints of multimodal fusion: case study in Weakly-Supervised Audio-Visual Video Parsing
• GEM: A General Evaluation Benchmark for Multimodal Tasks