Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey
arXiv:2111.01243
Abstract
Large, pre-trained transformer-based language models such as BERT have drastically changed the Natural Language Processing (NLP) field. We present a survey of recent work that uses these large language models to solve NLP tasks via pre-training then fine-tuning, prompting, or text generation approaches. We also present approaches that use pre-trained language models to generate data for training augmentation or other purposes. We conclude with discussions on limitations and suggested directions for future research.
References in corpus (22)
- Scaling Laws for Neural Language Models
- Evaluating Large Language Models Trained on Code
- The Pile: An 800GB Dataset of Diverse Text for Language Modeling
- What makes ImageNet good for transfer learning?
- Prefix-Tuning: Optimizing Continuous Prompts for Generation
- Assessing BERT's Syntactic Abilities
- Adaptation of Deep Bidirectional Multilingual Transformers for Russian Language
- BERTje: A Dutch BERT Model
- ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation
- Simple BERT Models for Relation Extraction and Semantic Role Labeling
- Multilingual is not enough: BERT for Finnish
- BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning
- Multimodal Few-Shot Learning with Frozen Language Models
- Zero-shot Text Classification With Generative Language Models
- Towards better substitution-based word sense induction
- XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoders
- AlephBERT:A Hebrew Large Pre-Trained Language Model to Start-off your Hebrew NLP Application With
- Learning Cross-Context Entity Representations from Text
- An Empirical Survey of Data Augmentation for Limited Data Learning in NLP
- On Data Augmentation for Extreme Multi-label Classification
- DARE: Data Augmented Relation Extraction with GPT-2
- Rewarding Smatch: Transition-Based AMR Parsing with Reinforcement Learning