AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing

Abstract

Transformer-based pretrained language models (T-PTLMs) have achieved great success in almost every NLP task. The evolution of these models started with GPT and BERT. These models are built on the top of transformers, self-supervised learning and transfer learning. Transformed-based PTLMs learn universal language representations from large volumes of text data using self-supervised learning and transfer this knowledge to downstream tasks. These models provide good background knowledge to downstream tasks which avoids training of downstream models from scratch. In this comprehensive survey paper, we initially give a brief overview of self-supervised learning. Next, we explain various core concepts like pretraining, pretraining methods, pretraining tasks, embeddings and downstream adaptation methods. Next, we present a new taxonomy of T-PTLMs and then give brief overview of various benchmarks including both intrinsic and extrinsic. We present a summary of various useful libraries to work with T-PTLMs. Finally, we highlight some of the future research directions which will further improve these models. We strongly believe that this comprehensive survey paper will serve as a good reference to learn the core concepts as well as to stay updated with the recent happenings in T-PTLMs.

arXiv commentPreprint under review

References in corpus (97)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Distilling the Knowledge in a Neural Network
• BioBERT: a pre-trained biomedical language representation model for biomedical text mining
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• A Survey on Visual Transformer
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Transformers in Vision: A Survey
• Language Models are Few-Shot Learners
• Longformer: The Long-Document Transformer
• Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
• Cross-lingual Language Model Pretraining
• Pre-trained Models for Natural Language Processing: A Survey
• Scaling Laws for Neural Language Models
• Do Deep Nets Really Need to be Deep?
• A Survey on Multi-view Learning
• SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
• CamemBERT: a Tasty French Language Model
• MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
• Multilingual Denoising Pre-training for Neural Machine Translation
• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better
• Q8BERT: Quantized 8Bit BERT
• DeBERTa: Decoding-enhanced BERT with Disentangled Attention
• CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
• Reformer: The Efficient Transformer
• BARTScore: Evaluating Generated Text as Text Generation
• XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• Extracting Training Data from Large Language Models
• Adaptation of Deep Bidirectional Multilingual Transformers for Russian Language
• Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks
• BERTje: A Dutch BERT Model
• Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping
• FinBERT: A Pretrained Language Model for Financial Communications
• GraphCodeBERT: Pre-training Code Representations with Data Flow
• MuRIL: Multilingual Representations for Indian Languages
• FinBERT: Financial Sentiment Analysis with Pre-trained Language Models
• CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation
• Making Pre-trained Language Models Better Few-shot Learners
• Towards Understanding and Mitigating Social Biases in Language Models
• Multilingual is not enough: BERT for Finnish
• Training with Quantization Noise for Extreme Model Compression
• BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning
• Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Models
• StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding
• DialoGLUE: A Natural Language Understanding Benchmark for Task-Oriented Dialogue
• PanGu-$\alpha$: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation
• CharBERT: Character-aware Pre-trained Language Model
• Charformer: Fast Character Transformers via Gradient-based Subword Tokenization
• Multi-task learning for natural language processing in the 2020s: where are we going?
• Adv-BERT: BERT is not robust on misspellings! Generating nature adversarial samples on BERT
• KLUE: Korean Language Understanding Evaluation
• InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
• PMIndia -- A Collection of Parallel Corpora of Languages of India
• CLUE: A Chinese Language Understanding Evaluation Benchmark
• Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning
• Conceptualized Representation Learning for Chinese Biomedical Text Mining
• MathBERT: A Pre-Trained Model for Mathematical Formula Understanding
• The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics
• ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
• Are Sixteen Heads Really Better than One?
• TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learning
• RobeCzech: Czech RoBERTa, a monolingual contextualized language representation model
• CLUECorpus2020: A Large-scale Chinese Corpus for Pre-training Language Model
• MT-BioNER: Multi-task Learning for Biomedical Named Entity Recognition using Deep Bidirectional Transformers
• Playing with Words at the National Library of Sweden -- Making a Swedish BERT
• TextBrewer: An Open-Source Knowledge Distillation Toolkit for Natural Language Processing
• FewCLUE: A Chinese Few-shot Learning Evaluation Benchmark
• AraELECTRA: Pre-Training Text Discriminators for Arabic Language Understanding
• A Benchmark for Lease Contract Review
• Processing South Asian Languages Written in the Latin Script: the Dakshina Dataset
• Transferability of Natural Language Inference to Biomedical Question Answering
• ExplainaBoard: An Explainable Leaderboard for NLP
• CausalBERT: Injecting Causal Knowledge Into Pre-trained Models with Minimal Supervision
• PTT5: Pretraining and validating the T5 model on Brazilian Portuguese data
• WangchanBERTa: Pretraining transformer-based Thai Language Models
• GLGE: A New General Language Generation Evaluation Benchmark
• Self-Supervised Learning from Contrastive Mixtures for Personalized Speech Enhancement
• CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddings
• CoTexT: Multi-task Learning with Code-Text Transformer
• ParsiNLU: A Suite of Language Understanding Challenges for Persian
• FLEX: Unifying Evaluation for Few-Shot NLP
• Domain-Relevant Embeddings for Medical Question Similarity
• Low Resource Multi-Task Sequence Tagging -- Revisiting Dynamic Conditional Random Fields
• Interpretable bias mitigation for textual data: Reducing gender bias in patient notes while maintaining classification performance
• Efficient pre-training objectives for Transformers
• Extending Multilingual BERT to Low-Resource Languages
• Common Sense Beyond English: Evaluating and Improving Multilingual Language Models for Commonsense Reasoning
• Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains
• Does He Wink or Does He Nod? A Challenging Benchmark for Evaluating Word Understanding of Language Models
• KD-Lib: A PyTorch library for Knowledge Distillation, Pruning and Quantization
• CLINE: Contrastive Learning with Semantic Negative Examples for Natural Language Understanding
• Dual-View Distilled BERT for Sentence Embedding
• FastSeq: Make Sequence Generation Faster

Cited by in corpus (4)

• Survey of different Large Language Model Architectures: Trends, Benchmarks, and Challenges
• Neuromorphic Camera Denoising using Graph Neural Network-driven Transformers
• Language Models as a Knowledge Source for Cognitive Agents
• An Overview on Generative AI at Scale with Edge-Cloud Computing