Pre-Trained Models: Past, Present and Future

Abstract

Large-scale pre-trained models (PTMs) such as BERT and GPT have recently achieved great success and become a milestone in the field of artificial intelligence (AI). Owing to sophisticated pre-training objectives and huge model parameters, large-scale PTMs can effectively capture knowledge from massive labeled and unlabeled data. By storing knowledge into huge parameters and fine-tuning on specific tasks, the rich knowledge implicitly encoded in huge parameters can benefit a variety of downstream tasks, which has been extensively demonstrated via experimental verification and empirical analysis. It is now the consensus of the AI community to adopt PTMs as backbone for downstream tasks rather than learning models from scratch. In this paper, we take a deep look into the history of pre-training, especially its special relation with transfer learning and self-supervised learning, to reveal the crucial position of PTMs in the AI development spectrum. Further, we comprehensively review the latest breakthroughs of PTMs. These breakthroughs are driven by the surge of computational power and the increasing availability of data, towards four important directions: designing effective architectures, utilizing rich contexts, improving computational efficiency, and conducting interpretation and theoretical analysis. Finally, we discuss a series of open problems and research directions of PTMs, and hope our view can inspire and advance the future study of PTMs.

References in corpus (93)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Distilling the Knowledge in a Neural Network
• BioBERT: a pre-trained biomedical language representation model for biomedical text mining
• A Simple Framework for Contrastive Learning of Visual Representations
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Longformer: The Long-Document Transformer
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Reconciling modern machine learning practice and the bias-variance trade-off
• Scaling Laws for Neural Language Models
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• Zero-Shot Text-to-Image Generation
• Deep Learning with Limited Numerical Precision
• VQA: Visual Question Answering
• Mixed Precision Training
• The Cityscapes Dataset for Semantic Urban Scene Understanding
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• Linformer: Self-Attention with Linear Complexity
• Domain Adaptation for Statistical Classifiers
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• ERNIE: Enhanced Representation through Knowledge Integration
• Dynamic Memory Networks for Visual and Textual Question Answering
• REALM: Retrieval-Augmented Language Model Pre-Training
• Large Batch Training of Convolutional Networks
• Layer Normalization
• Generating Long Sequences with Sparse Transformers
• A Convolutional Neural Network for Modelling Sentences
• Exploring Simple Siamese Representation Learning
• CogView: Mastering Text-to-Image Generation via Transformers
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks
• Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
• Assessing BERT's Syntactic Abilities
• Reducing Transformer Depth on Demand with Structured Dropout
• Big Bird: Transformers for Longer Sequences
• Towards a Human-like Open-Domain Chatbot
• Model compression via distillation and quantization
• Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answering
• Learned in Translation: Contextualized Word Vectors
• LXMERT: Learning Cross-Modality Encoder Representations from Transformers
• ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data
• VideoBERT: A Joint Model for Video and Language Representation Learning
• XNLI: Evaluating Cross-lingual Sentence Representations
• Supporting Very Large Models using Automatic Dataflow Graph Partitioning
• Beyond Data and Model Parallelism for Deep Neural Networks
• How multilingual is Multilingual BERT?
• K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
• TinyBERT: Distilling BERT for Natural Language Understanding
• Language Models as Knowledge Bases?
• Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
• PanGu-$\alpha$: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation
• NEZHA: Neural Contextualized Representation for Chinese Language Understanding
• WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Training
• Language Models are Open Knowledge Graphs
• InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
• ERNIE 2.0: A Continual Pre-training Framework for Language Understanding
• Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
• ZeRO-Offload: Democratizing Billion-Scale Model Training
• ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning
• Commonsense Knowledge Mining from Pretrained Models
• Knowledge-Aware Language Model Pretraining
• M3P: Learning Universal Representations via Multitask Multilingual Multimodal Pre-training
• FastMoE: A Fast Mixture-of-Expert Training System
• ZhuSuan: A Library for Bayesian Deep Learning
• When BERT Plays the Lottery, All Tickets Are Winning
• TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models
• Probing Neural Network Comprehension of Natural Language Arguments
• SpeechBERT: An Audio-and-text Jointly Learned Language Model for End-to-end Spoken Question Answering
• Can neural networks acquire a structural bias from raw linguistic data?
• Cognitive Graph for Multi-Hop Reading Comprehension at Scale
• ETC: Encoding Long and Structured Inputs in Transformers
• PTR: Prompt Tuning with Rules for Text Classification
• Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model
• CPM: A Large-scale Generative Chinese Pre-trained Language Model
• PALM: Pre-training an Autoencoding&Autoregressive Language Model for Context-conditioned Generation
• XGPT: Cross-modal Generative Pre-Training for Image Captioning
• CoLAKE: Contextualized Language and Knowledge Embedding
• Better Robustness by More Coverage: Adversarial Training with Mixup Augmentation for Robust Fine-tuning
• Inducing Syntactic Trees from BERT Representations
• Further Analysis of Outlier Detection with Deep Generative Models
• TernaryBERT: Distillation-aware Ultra-low Bit BERT
• Generating Adversarial Examples in Chinese Texts Using Sentence-Pieces
• Variance-reduced Language Pretraining via a Mask Proposal Network
• Understanding Multi-Head Attention in Abstractive Summarization
• Reasoning Over Virtual Knowledge Bases With Open Predicate Relations
• BERT's output layer recognizes all hidden layers? Some Intriguing Phenomena and a simple way to boost BERT
• Parsing as Pretraining
• Train No Evil: Selective Masking for Task-Guided Pre-Training
• Adversarial Language Games for Advanced Natural Language Intelligence
• Towards a Universal Continuous Knowledge Base

Cited by in corpus (2)

• BERT-like Pre-training for Symbolic Piano Music Classification Tasks
• CoEdPilot: Recommending Code Edits with Learned Prior Edit Relevance, Project-wise Awareness, and Interactive Nature