AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing
arXiv:2108.05542
Abstract
Transformer-based pretrained language models (T-PTLMs) have achieved great success in almost every NLP task. The evolution of these models started with GPT and BERT. These models are built on the top of transformers, self-supervised learning and transfer learning. Transformed-based PTLMs learn universal language representations from large volumes of text data using self-supervised learning and transfer this knowledge to downstream tasks. These models provide good background knowledge to downstream tasks which avoids training of downstream models from scratch. In this comprehensive survey paper, we initially give a brief overview of self-supervised learning. Next, we explain various core concepts like pretraining, pretraining methods, pretraining tasks, embeddings and downstream adaptation methods. Next, we present a new taxonomy of T-PTLMs and then give brief overview of various benchmarks including both intrinsic and extrinsic. We present a summary of various useful libraries to work with T-PTLMs. Finally, we highlight some of the future research directions which will further improve these models. We strongly believe that this comprehensive survey paper will serve as a good reference to learn the core concepts as well as to stay updated with the recent happenings in T-PTLMs.
Preprint under review
References in corpus (61)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Distilling the Knowledge in a Neural Network
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Language Models are Few-Shot Learners
- Cross-lingual Language Model Pretraining
- Scaling Laws for Neural Language Models
- A Survey on Multi-view Learning
- Multilingual Denoising Pre-training for Neural Machine Translation
- MASS: Masked Sequence to Sequence Pre-training for Language Generation
- Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better
- CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
- GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
- Reformer: The Efficient Transformer
- BARTScore: Evaluating Generated Text as Text Generation
- Prefix-Tuning: Optimizing Continuous Prompts for Generation
- Adaptation of Deep Bidirectional Multilingual Transformers for Russian Language
- BERTje: A Dutch BERT Model
- Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping
- FinBERT: A Pretrained Language Model for Financial Communications
- MuRIL: Multilingual Representations for Indian Languages
- CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation
- Towards Understanding and Mitigating Social Biases in Language Models
- Multilingual is not enough: BERT for Finnish
- BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning
- StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding
- DialoGLUE: A Natural Language Understanding Benchmark for Task-Oriented Dialogue
- PanGu-: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation
- Charformer: Fast Character Transformers via Gradient-based Subword Tokenization
- Multi-task learning for natural language processing in the 2020s: where are we going?
- KLUE: Korean Language Understanding Evaluation
- PMIndia -- A Collection of Parallel Corpora of Languages of India
- Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning
- MathBERT: A Pre-Trained Model for Mathematical Formula Understanding
- The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics
- ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
- TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learning
- RobeCzech: Czech RoBERTa, a monolingual contextualized language representation model
- CLUECorpus2020: A Large-scale Chinese Corpus for Pre-training Language Model
- MT-BioNER: Multi-task Learning for Biomedical Named Entity Recognition using Deep Bidirectional Transformers
- Playing with Words at the National Library of Sweden -- Making a Swedish BERT
- FewCLUE: A Chinese Few-shot Learning Evaluation Benchmark
- AraELECTRA: Pre-Training Text Discriminators for Arabic Language Understanding
- Processing South Asian Languages Written in the Latin Script: the Dakshina Dataset
- ExplainaBoard: An Explainable Leaderboard for NLP
- CausalBERT: Injecting Causal Knowledge Into Pre-trained Models with Minimal Supervision
- WangchanBERTa: Pretraining transformer-based Thai Language Models
- CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddings
- CoTexT: Multi-task Learning with Code-Text Transformer
- FLEX: Unifying Evaluation for Few-Shot NLP
- Domain-Relevant Embeddings for Medical Question Similarity
- Efficient pre-training objectives for Transformers
- Interpretable bias mitigation for textual data: Reducing gender bias in patient notes while maintaining classification performance
- Low Resource Multi-Task Sequence Tagging -- Revisiting Dynamic Conditional Random Fields
- Extending Multilingual BERT to Low-Resource Languages
- KD-Lib: A PyTorch library for Knowledge Distillation, Pruning and Quantization
- Common Sense Beyond English: Evaluating and Improving Multilingual Language Models for Commonsense Reasoning
- Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains
- Does He Wink or Does He Nod? A Challenging Benchmark for Evaluating Word Understanding of Language Models
- FastSeq: Make Sequence Generation Faster
- CLINE: Contrastive Learning with Semantic Negative Examples for Natural Language Understanding
- Dual-View Distilled BERT for Sentence Embedding