Pre-trained Models for Natural Language Processing: A Survey

Abstract

Recently, the emergence of pre-trained models (PTMs) has brought natural language processing (NLP) to a new era. In this survey, we provide a comprehensive review of PTMs for NLP. We first briefly introduce language representation learning and its research progress. Then we systematically categorize existing PTMs based on a taxonomy with four perspectives. Next, we describe how to adapt the knowledge of PTMs to the downstream tasks. Finally, we outline some potential directions of PTMs for future research. This survey is purposed to be a hands-on guide for understanding, using, and developing PTMs for various NLP tasks.

arXiv commentInvited Review of Science China Technological Sciences

References in corpus (104)

• Distributed Representations of Words and Phrases and their Compositionality
• Distilling the Knowledge in a Neural Network
• Sequence to Sequence Learning with Neural Networks
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Semi-Supervised Classification with Graph Convolutional Networks
• BioBERT: a pre-trained biomedical language representation model for biomedical text mining
• Natural Language Processing (almost) from Scratch
• Distributed Representations of Sentences and Documents
• Neural Architecture Search with Reinforcement Learning
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Convolutional Sequence to Sequence Learning
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• Learning deep representations by mutual information estimation and maximization
• ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
• Cross-lingual Language Model Pretraining
• Pre-trained Models for Natural Language Processing: A Survey
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• Recurrent Neural Network for Text Classification with Multi-Task Learning
• Character-Aware Neural Language Models
• SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
• Unified Language Model Pre-training for Natural Language Understanding and Generation
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• CTRL: A Conditional Transformer Language Model for Controllable Generation
• VL-BERT: Pre-training of Generic Visual-Linguistic Representations
• ERNIE: Enhanced Representation through Knowledge Integration
• CamemBERT: a Tasty French Language Model
• Publicly Available Clinical BERT Embeddings
• Skip-Thought Vectors
• AraBERT: Transformer-based Model for Arabic Language Understanding
• MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
• ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission
• Multilingual Denoising Pre-training for Neural Machine Translation
• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• Attention is not Explanation
• Semi-supervised Sequence Learning
• Q8BERT: Quantized 8Bit BERT
• Well-Read Students Learn Better: On the Importance of Pre-training Compact Models
• Universal Transformers
• BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysis
• Adaptive Computation Time for Recurrent Neural Networks
• Utilizing BERT for Aspect-Based Sentiment Analysis via Constructing Auxiliary Sentence
• Distilling Task-Specific Knowledge from BERT into Simple Neural Networks
• Learning How to Ask: Querying LMs with Mixtures of Soft Prompts
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• Assessing BERT's Syntactic Abilities
• FreeLB: Enhanced Adversarial Training for Natural Language Understanding
• What do Neural Machine Translation Models Learn about Morphology?
• Adaptation of Deep Bidirectional Multilingual Transformers for Russian Language
• Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks
• Learned in Translation: Contextualized Word Vectors
• UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
• Learning Video Representations using Contrastive Bidirectional Transformer
• BERTje: A Dutch BERT Model
• Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping
• A Theoretical Analysis of Contrastive Unsupervised Representation Learning
• UNITER: UNiversal Image-TExt Representation Learning
• Incorporating BERT into Neural Machine Translation
• Cross-Lingual Ability of Multilingual BERT: An Empirical Study
• Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding
• Parameter-Efficient Transfer Learning for NLP
• What do you learn from context? Probing for sentence structure in contextualized word representations
• Data Augmentation using Pre-trained Transformer Models
• K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
• TinyBERT: Distilling BERT for Natural Language Understanding
• Adapt or Get Left Behind: Domain Adaptation through BERT Language Model Finetuning for Aspect-Target Sentiment Classification
• Making Pre-trained Language Models Better Few-shot Learners
• Multilingual is not enough: BERT for Finnish
• Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training
• BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning
• StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding
• MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
• Adversarial Training for Large Neural Language Models
• PatentBERT: Patent Classification with Fine-Tuning a pre-trained BERT Model
• NEZHA: Neural Contextualized Representation for Chinese Language Understanding
• A Mutual Information Maximization Perspective of Language Representation Learning
• Adv-BERT: BERT is not robust on misspellings! Generating nature adversarial samples on BERT
• KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation
• BERT-ATTACK: Adversarial Attack Against BERT Using BERT
• FlauBERT: Unsupervised Language Model Pre-training for French
• Utilizing BERT Intermediate Layers for Aspect Based Sentiment Analysis and Natural Language Inference
• FastBERT: a Self-distilling BERT with Adaptive Inference Time
• exBERT: A Visual Analysis Tool to Explore Learned Representations in Transformers Models
• Exploiting BERT for End-to-End Aspect-based Sentiment Analysis
• BERT Loses Patience: Fast and Robust Inference with Early Exit
• Commonsense Knowledge Mining from Pretrained Models
• Technical report on Conversational Question Answering
• Integrating Graph Contextualized Knowledge into Pre-trained Language Models
• Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Induction
• BERT-of-Theseus: Compressing BERT by Progressive Module Replacing
• Progress Notes Classification and Keyword Extraction using Attention-based Deep Learning Models with BERT
• TextBrewer: An Open-Source Knowledge Distillation Toolkit for Natural Language Processing
• TwinBERT: Distilling Knowledge to Twin-Structured BERT Models for Efficient Retrieval
• SpeechBERT: An Audio-and-text Jointly Learned Language Model for End-to-end Spoken Question Answering
• Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model
• Factual Probing Is [MASK]: Learning vs. Learning to Recall
• Transformer to CNN: Label-scarce distillation for efficient text classification
• A Knowledge-Enhanced Pretraining Model for Commonsense Story Generation
• "Mask and Infill" : Applying Masked Language Model to Sentiment Transfer
• Early Exiting with Ensemble Internal Classifiers
• ZEN: Pre-training Chinese Text Encoder Enhanced by N-gram Representations
• Efficient Contextualized Representation: Language Model Pruning for Sequence Labeling

Cited by in corpus (133)

• A Survey on Visual Transformer
• Pre-trained Models for Natural Language Processing: A Survey
• Unifying Large Language Models and Knowledge Graphs: A Roadmap
• Self-supervised Learning: Generative or Contrastive
• Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Users
• A Survey of Human-in-the-loop for Machine Learning
• FedKD: Communication Efficient Federated Learning via Knowledge Distillation
• EEG based Emotion Recognition: A Tutorial and Review
• Deep Learning Based Text Classification: A Comprehensive Review
• Self-Supervised Speech Representation Learning: A Review
• Pre-training Enhanced Spatial-temporal Graph Neural Network for Multivariate Time Series Forecasting
• Towards a Unified View of Parameter-Efficient Transfer Learning
• VLP: A Survey on Vision-Language Pre-training
• A Survey of Knowledge-Enhanced Text Generation
• Natural Language Processing for Smart Healthcare
• Controllable Protein Design with Language Models
• Graph Neural Networks: Taxonomy, Advances and Trends
• PalmTree: Learning an Assembly Language Model for Instruction Embedding
• Compressing Large-Scale Transformer-Based Models: A Case Study on BERT
• ConvBERT: Improving BERT with Span-based Dynamic Convolution
• Machine Knowledge: Creation and Curation of Comprehensive Knowledge Bases
• CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation
• The Impact of Multiple Parallel Phrase Suggestions on Email Input and Composition Behaviour of Native and Non-Native English Writers
• Deep Learning for Android Malware Defenses: a Systematic Literature Review
• Paradigm Shift in Natural Language Processing
• Does the Magic of BERT Apply to Medical Code Assignment? A Quantitative Study
• Cluster-Level Contrastive Learning for Emotion Recognition in Conversations
• Backdoor Pre-trained Models Can Transfer to All
• Linguistically inspired roadmap for building biologically reliable protein language models
• LC-LLM: Explainable Lane-Change Intention and Trajectory Predictions with Large Language Models
• A Review of Predictive and Contrastive Self-supervised Learning for Medical Images
• A Survey on Multilingual Large Language Models: Corpora, Alignment, and Bias
• CTRAN: CNN-Transformer-based Network for Natural Language Understanding
• Mask-guided BERT for Few Shot Text Classification
• Model Pruning Enables Localized and Efficient Federated Learning for Yield Forecasting and Data Sharing
• PTM4Tag: Sharpening Tag Recommendation of Stack Overflow Posts with Pre-trained Models
• FoodSAM: Any Food Segmentation
• Distributed Deep Reinforcement Learning: A Survey and A Multi-Player Multi-Agent Learning Toolbox
• AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing
• GANSlider: How Users Control Generative Models for Images using Multiple Sliders with and without Feedforward Information
• Achieving Peak Performance for Large Language Models: A Systematic Review
• Natural Language Interfaces to Data
• AI in Human-computer Gaming: Techniques, Challenges and Opportunities
• Automated scholarly paper review: Concepts, technologies, and challenges
• Pre-trained Language Models in Biomedical Domain: A Systematic Survey
• Lightweight Transformers for Clinical Natural Language Processing
• Are Pre-trained Convolutions Better than Pre-trained Transformers?
• LogPrécis: Unleashing Language Models for Automated Malicious Log Analysis
• Graph Foundation Models: Concepts, Opportunities and Challenges
• Gender Bias in BERT -- Measuring and Analysing Biases through Sentiment Rating in a Realistic Downstream Classification Task
• A Benchmark for Lease Contract Review
• PTR: Prompt Tuning with Rules for Text Classification
• Training-free Lexical Backdoor Attacks on Language Models
• Knowledge Enhanced Pretrained Language Models: A Compreshensive Survey
• A Survey of Knowledge Enhanced Pre-trained Models
• Mask and Cloze: Automatic Open Cloze Question Generation using a Masked Language Model
• Disentangled Variational Autoencoder for Emotion Recognition in Conversations
• Unstructured Text Enhanced Open-domain Dialogue System: A Systematic Survey
• CoLAKE: Contextualized Language and Knowledge Embedding
• More but Correct: Generating Diversified and Entity-revised Medical Response
• Is Your Model Sensitive? SPeDaC: A New Benchmark for Detecting and Classifying Sensitive Personal Data
• Pretrained Language Models for Text Generation: A Survey
• Artificial intelligence is algorithmic mimicry: why artificial "agents" are not (and won't be) proper agents
• Early Exiting with Ensemble Internal Classifiers
• The Life Cycle of Knowledge in Big Language Models: A Survey
• NodePiece: Compositional and Parameter-Efficient Representations of Large Knowledge Graphs
• Towards an Enhanced Understanding of Bias in Pre-trained Neural Language Models: A Survey with Special Emphasis on Affective Bias
• Explainable Recommendation with Personalized Review Retrieval and Aspect Learning
• A Multimodal Late Fusion Model for E-Commerce Product Classification
• Morphosyntactic probing of multilingual BERT models
• A Unified Generative Framework for Aspect-Based Sentiment Analysis
• Multimodal Side-Tuning for Document Classification
• Ranking and Tuning Pre-trained Models: A New Paradigm for Exploiting Model Hubs
• On the Effectiveness of Pretrained Models for API Learning
• SCARF: Self-Supervised Contrastive Learning using Random Feature Corruption
• PTUM: Pre-training User Model from Unlabeled User Behaviors via Self-supervision
• Infusing Disease Knowledge into BERT for Health Question Answering, Medical Inference and Disease Name Recognition
• Literature review on vulnerability detection using NLP technology
• Emergent Language: A Survey and Taxonomy
• Scattered or Connected? An Optimized Parameter-efficient Tuning Approach for Information Retrieval
• Learning Bidirectional Translation between Descriptions and Actions with Small Paired Data
• Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks
• Accurate, yet inconsistent? Consistency Analysis on Language Understanding Models
• Self-Supervised Learning for Text Recognition: A Critical Survey
• Encoder-Decoder Models Can Benefit from Pre-trained Masked Language Models in Grammatical Error Correction
• Exploring Motivations for Algorithm Mention in the Domain of Natural Language Processing: A Deep Learning Approach
• A Large-Scale Chinese Short-Text Conversation Dataset
• LRC-BERT: Latent-representation Contrastive Knowledge Distillation for Natural Language Understanding
• Empowering News Recommendation with Pre-trained Language Models
• Machine Learning for Multimodal Electronic Health Records-based Research: Challenges and Perspectives
• Exploring the Innovation Opportunities for Pre-trained Models
• A Unified Generative Framework for Various NER Subtasks
• Enhancing Function Name Prediction using Votes-Based Name Tokenization and Multi-Task Learning
• Trained Without My Consent: Detecting Code Inclusion In Language Models Trained on Code
• Enhancing Source Code Classification Effectiveness via Prompt Learning Incorporating Knowledge Features
• Accelerating BERT Inference for Sequence Labeling via Early-Exit
• One Teacher is Enough? Pre-trained Language Model Distillation from Multiple Teachers
• An Interdisciplinary Perspective on Evaluation and Experimental Design for Visual Text Analytics: Position Paper
• Attentive Representation Learning with Adversarial Training for Short Text Clustering
• Low-Resource Multi-Granularity Academic Function Recognition Based on Multiple Prompt Knowledge
• Transfer Learning for Multi-lingual Tasks -- a Survey
• Beyond Pixels: Leveraging the Language of Soccer to Improve Spatio-Temporal Action Detection in Broadcast Videos
• Approximating Human Strategic Reasoning with LLM-Enhanced Recursive Reasoners Leveraging Multi-agent Hypergames
• LARR: Large Language Model Aided Real-time Scene Recommendation with Semantic Understanding
• NLP is Not enough -- Contextualization of User Input in Chatbots
• NPS-AntiClone: Identity Cloning Detection based on Non-Privacy-Sensitive User Profile Data
• NewsBERT: Distilling Pre-trained Language Model for Intelligent News Application
• A Graph Representation of Semi-structured Data for Web Question Answering
• Knowledge Transfer via Pre-training for Recommendation: A Review and Prospect
• The meaning of prompts and the prompts of meaning: Semiotic reflections and modelling
• SimCLAD: A Simple Framework for Contrastive Learning of Acronym Disambiguation
• fastHan: A BERT-based Multi-Task Toolkit for Chinese NLP
• Backdoor Attacks on Pre-trained Models by Layerwise Weight Poisoning
• CCAE: A Corpus of Chinese-based Asian Englishes
• SMedBERT: A Knowledge-Enhanced Pre-trained Language Model with Structured Semantics for Medical Text Mining
• Human Activity Recognition in an Open World
• A Survey on Green Deep Learning
• Discriminative, Generative and Self-Supervised Approaches for Target-Agnostic Learning
• Weight Squeezing: Reparameterization for Knowledge Transfer and Model Compression
• Pre-training and Diagnosing Knowledge Base Completion Models
• Large Models for Battery Prognostics and Health Management: A Review and Future Roadmap
• Want to Identify, Extract and Normalize Adverse Drug Reactions in Tweets? Use RoBERTa
• Position-based Contributive Embeddings for Aspect-Based Sentiment Analysis
• Transformer Scalability Crisis: The First Comprehensive Empirical Analysis of Performance Walls in Modern Language Models
• Multilingual Translation via Grafting Pre-trained Language Models
• Navigating the Kaleidoscope of COVID-19 Misinformation Using Deep Learning
• Efficient Attribute Injection for Pretrained Language Models
• Cross-lingual Intermediate Fine-tuning improves Dialogue State Tracking
• A Concise Model for Multi-Criteria Chinese Word Segmentation with Transformer Encoder
• TCube: Domain-Agnostic Neural Time-series Narration
• Event-Driven Learning of Systematic Behaviours in Stock Markets
• REPT: Bridging Language Models and Machine Reading Comprehension via Retrieval-Based Pre-training
• Scalability vs. Utility: Do We Have to Sacrifice One for the Other in Data Importance Quantification?