Pre-Trained Models: Past, Present and Future

Abstract

Large-scale pre-trained models (PTMs) such as BERT and GPT have recently achieved great success and become a milestone in the field of artificial intelligence (AI). Owing to sophisticated pre-training objectives and huge model parameters, large-scale PTMs can effectively capture knowledge from massive labeled and unlabeled data. By storing knowledge into huge parameters and fine-tuning on specific tasks, the rich knowledge implicitly encoded in huge parameters can benefit a variety of downstream tasks, which has been extensively demonstrated via experimental verification and empirical analysis. It is now the consensus of the AI community to adopt PTMs as backbone for downstream tasks rather than learning models from scratch. In this paper, we take a deep look into the history of pre-training, especially its special relation with transfer learning and self-supervised learning, to reveal the crucial position of PTMs in the AI development spectrum. Further, we comprehensively review the latest breakthroughs of PTMs. These breakthroughs are driven by the surge of computational power and the increasing availability of data, towards four important directions: designing effective architectures, utilizing rich contexts, improving computational efficiency, and conducting interpretation and theoretical analysis. Finally, we discuss a series of open problems and research directions of PTMs, and hope our view can inspire and advance the future study of PTMs.

References in corpus (40)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Distilling the Knowledge in a Neural Network
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Scaling Laws for Neural Language Models
• Zero-Shot Text-to-Image Generation
• Deep Learning with Limited Numerical Precision
• Linformer: Self-Attention with Linear Complexity
• ERNIE: Enhanced Representation through Knowledge Integration
• REALM: Retrieval-Augmented Language Model Pre-Training
• Large Batch Training of Convolutional Networks
• Layer Normalization
• Generating Long Sequences with Sparse Transformers
• A Convolutional Neural Network for Modelling Sentences
• Exploring Simple Siamese Representation Learning
• Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
• Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks
• Assessing BERT's Syntactic Abilities
• Reducing Transformer Depth on Demand with Structured Dropout
• Towards a Human-like Open-Domain Chatbot
• ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data
• How multilingual is Multilingual BERT?
• PanGu-$\alpha$: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation
• Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
• ZeRO-Offload: Democratizing Billion-Scale Model Training
• ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning
• FastMoE: A Fast Mixture-of-Expert Training System
• ZhuSuan: A Library for Bayesian Deep Learning
• Can neural networks acquire a structural bias from raw linguistic data?
• Cognitive Graph for Multi-Hop Reading Comprehension at Scale
• Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model
• CPM: A Large-scale Generative Chinese Pre-trained Language Model
• XGPT: Cross-modal Generative Pre-Training for Image Captioning
• CoLAKE: Contextualized Language and Knowledge Embedding
• Inducing Syntactic Trees from BERT Representations
• Understanding Multi-Head Attention in Abstractive Summarization
• Generating Adversarial Examples in Chinese Texts Using Sentence-Pieces
• Variance-reduced Language Pretraining via a Mask Proposal Network
• Parsing as Pretraining
• Towards a Universal Continuous Knowledge Base