AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing

Abstract

Transformer-based pretrained language models (T-PTLMs) have achieved great success in almost every NLP task. The evolution of these models started with GPT and BERT. These models are built on the top of transformers, self-supervised learning and transfer learning. Transformed-based PTLMs learn universal language representations from large volumes of text data using self-supervised learning and transfer this knowledge to downstream tasks. These models provide good background knowledge to downstream tasks which avoids training of downstream models from scratch. In this comprehensive survey paper, we initially give a brief overview of self-supervised learning. Next, we explain various core concepts like pretraining, pretraining methods, pretraining tasks, embeddings and downstream adaptation methods. Next, we present a new taxonomy of T-PTLMs and then give brief overview of various benchmarks including both intrinsic and extrinsic. We present a summary of various useful libraries to work with T-PTLMs. Finally, we highlight some of the future research directions which will further improve these models. We strongly believe that this comprehensive survey paper will serve as a good reference to learn the core concepts as well as to stay updated with the recent happenings in T-PTLMs.

Preprint under review

References in corpus (53)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Language Models are Few-Shot Learners
• Cross-lingual Language Model Pretraining
• Scaling Laws for Neural Language Models
• A Survey on Multi-view Learning
• Multilingual Denoising Pre-training for Neural Machine Translation
• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
• Reformer: The Efficient Transformer
• BARTScore: Evaluating Generated Text as Text Generation
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• Adaptation of Deep Bidirectional Multilingual Transformers for Russian Language
• BERTje: A Dutch BERT Model
• Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping
• FinBERT: A Pretrained Language Model for Financial Communications
• MuRIL: Multilingual Representations for Indian Languages
• Towards Understanding and Mitigating Social Biases in Language Models
• Multilingual is not enough: BERT for Finnish
• BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning
• StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding
• DialoGLUE: A Natural Language Understanding Benchmark for Task-Oriented Dialogue
• PanGu-$\alpha$: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation
• Multi-task learning for natural language processing in the 2020s: where are we going?
• PMIndia -- A Collection of Parallel Corpora of Languages of India
• Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning
• The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics
• ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
• MT-BioNER: Multi-task Learning for Biomedical Named Entity Recognition using Deep Bidirectional Transformers
• CLUECorpus2020: A Large-scale Chinese Corpus for Pre-training Language Model
• Playing with Words at the National Library of Sweden -- Making a Swedish BERT
• FewCLUE: A Chinese Few-shot Learning Evaluation Benchmark
• AraELECTRA: Pre-Training Text Discriminators for Arabic Language Understanding
• Processing South Asian Languages Written in the Latin Script: the Dakshina Dataset
• CausalBERT: Injecting Causal Knowledge Into Pre-trained Models with Minimal Supervision
• WangchanBERTa: Pretraining transformer-based Thai Language Models
• CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddings
• CoTexT: Multi-task Learning with Code-Text Transformer
• FLEX: Unifying Evaluation for Few-Shot NLP
• Domain-Relevant Embeddings for Medical Question Similarity
• Interpretable bias mitigation for textual data: Reducing gender bias in patient notes while maintaining classification performance
• Efficient pre-training objectives for Transformers
• Low Resource Multi-Task Sequence Tagging -- Revisiting Dynamic Conditional Random Fields
• Extending Multilingual BERT to Low-Resource Languages
• Does He Wink or Does He Nod? A Challenging Benchmark for Evaluating Word Understanding of Language Models
• Common Sense Beyond English: Evaluating and Improving Multilingual Language Models for Commonsense Reasoning
• Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains
• KD-Lib: A PyTorch library for Knowledge Distillation, Pruning and Quantization
• Dual-View Distilled BERT for Sentence Embedding
• CLINE: Contrastive Learning with Semantic Negative Examples for Natural Language Understanding
• FastSeq: Make Sequence Generation Faster

Cited by in corpus (1)

• Language Models as a Knowledge Source for Cognitive Agents