M6: A Chinese Multimodal Pretrainer

Abstract

In this work, we construct the largest dataset for multimodal pretraining in Chinese, which consists of over 1.9TB images and 292GB texts that cover a wide range of domains. We propose a cross-modal pretraining method called M6, referring to Multi-Modality to Multi-Modality Multitask Mega-transformer, for unified pretraining on the data of single modality and multiple modalities. We scale the model size up to 10 billion and 100 billion parameters, and build the largest pretrained model in Chinese. We apply the model to a series of downstream applications, and demonstrate its outstanding performance in comparison with strong baselines. Furthermore, we specifically design a downstream task of text-guided image generation, and show that the finetuned M6 can create high-quality images with high resolution and abundant details.

arXiv comment12 pages, technical report. Extension of paper "M6" accepted to KDD 2021

References in corpus (22)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Language Models are Few-Shot Learners
• Neural Discrete Representation Learning
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• Zero-Shot Text-to-Image Generation
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• DeBERTa: Decoding-enhanced BERT with Disentangled Attention
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
• Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers
• Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answering
• ConvBERT: Improving BERT with Span-based Dynamic Convolution
• ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph
• Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training
• StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding
• BASE Layers: Simplifying Training of Large, Sparse Models
• InterBERT: Vision-and-Language Interaction for Multi-modal Pretraining
• CLUECorpus2020: A Large-scale Chinese Corpus for Pre-training Language Model
• FashionBERT: Text and Image Matching with Adaptive Loss for Cross-modal Retrieval

Cited by in corpus (13)

• FILIP: Fine-grained Interactive Language-Image Pre-Training
• ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation
• Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers
• Vector Quantized Diffusion Model for Text-to-Image Synthesis
• Fine-grained Visual-Text Prompt-Driven Self-Training for Open-Vocabulary Object Detection
• Yuan 1.0: Large-Scale Pre-trained Language Model in Zero-Shot and Few-Shot Learning
• M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining
• M6-T: Exploring Sparse Expert Models and Beyond
• Modeling Text-visual Mutual Dependency for Multi-modal Dialog Generation
• Connecting Language and Vision for Natural Language-Based Vehicle Retrieval
• Learning Relation Alignment for Calibrated Cross-modal Retrieval
• A Survey on Green Deep Learning
• On the Limitations of Multimodal VAEs