Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey

Abstract

Large, pre-trained transformer-based language models such as BERT have drastically changed the Natural Language Processing (NLP) field. We present a survey of recent work that uses these large language models to solve NLP tasks via pre-training then fine-tuning, prompting, or text generation approaches. We also present approaches that use pre-trained language models to generate data for training augmentation or other purposes. We conclude with discussions on limitations and suggested directions for future research.

References in corpus (56)

• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• Scaling Laws for Neural Language Models
• Evaluating Large Language Models Trained on Code
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission
• The Pile: An 800GB Dataset of Diverse Text for Language Modeling
• KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction
• BARTScore: Evaluating Generated Text as Text Generation
• A Simple Method for Commonsense Reasoning
• What makes ImageNet good for transfer learning?
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• Assessing BERT's Syntactic Abilities
• Adaptation of Deep Bidirectional Multilingual Transformers for Russian Language
• A Survey of Knowledge-Enhanced Text Generation
• CodeBERT: A Pre-Trained Model for Programming and Natural Languages
• BERTje: A Dutch BERT Model
• ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation
• True Few-Shot Learning with Language Models
• Simple BERT Models for Relation Extraction and Semantic Role Labeling
• Portuguese Named Entity Recognition using BERT-CRF
• A Simple Language Model for Task-Oriented Dialogue
• Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning
• Text Summarization with Pretrained Encoders
• Multilingual is not enough: BERT for Finnish
• BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning
• Entailment as Few-Shot Learner
• How to Fine-Tune BERT for Text Classification?
• Investigating Entity Knowledge in BERT with Simple Neural End-To-End Entity Linking
• Multimodal Few-Shot Learning with Frozen Language Models
• Zero-shot Text Classification With Generative Language Models
• Few-Shot Text Generation with Pattern-Exploiting Training
• Calibrate Before Use: Improving Few-Shot Performance of Language Models
• Finetuned Language Models Are Zero-Shot Learners
• Multi-domain Dialogue State Tracking as Dynamic Knowledge Graph Enhanced Question Answering
• BERT Rediscovers the Classical NLP Pipeline
• Towards better substitution-based word sense induction
• Commonsense Knowledge Mining from Pretrained Models
• PTR: Prompt Tuning with Rules for Text Classification
• AlephBERT:A Hebrew Large Pre-Trained Language Model to Start-off your Hebrew NLP Application With
• XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoders
• Parameter-Efficient Transfer Learning with Diff Pruning
• Factual Probing Is [MASK]: Learning vs. Learning to Recall
• Learning Cross-Context Entity Representations from Text
• Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
• An Empirical Survey of Data Augmentation for Limited Data Learning in NLP
• Probing and Fine-tuning Reading Comprehension Models for Few-shot Event Extraction
• Automated Concatenation of Embeddings for Structured Prediction
• On Data Augmentation for Extreme Multi-label Classification
• Second-Order Neural Dependency Parsing with Message Passing and End-to-End Training
• DARE: Data Augmented Relation Extraction with GPT-2
• AmbigQA: Answering Ambiguous Open-domain Questions
• Rewarding Smatch: Transition-Based AMR Parsing with Reinforcement Learning
• Does BERT agree? Evaluating knowledge of structure dependence through agreement relations
• Textual Explanations for Self-Driving Vehicles
• The Turking Test: Can Language Models Understand Instructions?
• Thank you BART! Rewarding Pre-Trained Models Improves Formality Style Transfer

Cited by in corpus (6)

• Deep Potentials for Materials Science
• Safety Analysis in the Era of Large Language Models: A Case Study of STPA using ChatGPT
• ContrastRepair: Enhancing Conversation-Based Automated Program Repair via Contrastive Test Case Pairs
• The Impact of Example Selection in Few-Shot Prompting on Automated Essay Scoring Using GPT Models
• Embedding generation for text classification of Brazilian Portuguese user reviews: from bag-of-words to transformers
• ImmunoLingo: Linguistics-based formalization of the antibody language