The NLP Cookbook: Modern Recipes for Transformer based Deep Learning Architectures

Abstract

In recent years, Natural Language Processing (NLP) models have achieved phenomenal success in linguistic and semantic tasks like text classification, machine translation, cognitive dialogue systems, information retrieval via Natural Language Understanding (NLU), and Natural Language Generation (NLG). This feat is primarily attributed due to the seminal Transformer architecture, leading to designs such as BERT, GPT (I, II, III), etc. Although these large-size models have achieved unprecedented performances, they come at high computational costs. Consequently, some of the recent NLP architectures have utilized concepts of transfer learning, pruning, quantization, and knowledge distillation to achieve moderate model sizes while keeping nearly similar performances as achieved by their predecessors. Additionally, to mitigate the data size challenge raised by language models from a knowledge extraction perspective, Knowledge Retrievers have been built to extricate explicit data documents from a large corpus of databases with greater efficiency and accuracy. Recent research has also focused on superior inference by providing efficient attention to longer input sequences. In this paper, we summarize and examine the current state-of-the-art (SOTA) NLP models that have been employed for numerous NLP tasks for optimal performance and efficiency. We provide a detailed understanding and functioning of the different architectures, a taxonomy of NLP designs, comparative evaluations, and future directions in NLP.

arXiv comment27 pages and 29 figures

References in corpus (46)

• Efficient Estimation of Word Representations in Vector Space
• Neural Machine Translation by Jointly Learning to Align and Translate
• Distilling the Knowledge in a Neural Network
• Sequence to Sequence Learning with Neural Networks
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• On the difficulty of training Recurrent Neural Networks
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Language Models are Few-Shot Learners
• Longformer: The Long-Document Transformer
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Cross-lingual Language Model Pretraining
• SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
• Rethinking the Value of Network Pruning
• Linformer: Self-Attention with Linear Complexity
• Benchmark Analysis of Representative Deep Neural Network Architectures
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• Ask Me Anything: Dynamic Memory Networks for Natural Language Processing
• Multilingual Denoising Pre-training for Neural Machine Translation
• MASS: Masked Sequence to Sequence Pre-training for Language Generation
• REALM: Retrieval-Augmented Language Model Pre-Training
• Generating Long Sequences with Sparse Transformers
• Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
• Q8BERT: Quantized 8Bit BERT
• Towards Accurate Binary Convolutional Neural Network
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
• Distilling Task-Specific Knowledge from BERT into Simple Neural Networks
• The Natural Language Decathlon: Multitask Learning as Question Answering
• Reformer: The Efficient Transformer
• Reducing Transformer Depth on Demand with Structured Dropout
• Big Bird: Transformers for Longer Sequences
• CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data
• Phrase-Based & Neural Unsupervised Machine Translation
• Know What You Don't Know: Unanswerable Questions for SQuAD
• The Reversible Residual Network: Backpropagation Without Storing Activations
• Exploring Sparsity in Recurrent Neural Networks
• TinyBERT: Distilling BERT for Natural Language Understanding
• Rethinking Attention with Performers
• HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
• Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
• LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks
• Answering Complex Open-domain Questions Through Iterative Query Generation
• Cross-Lingual BERT Transformation for Zero-Shot Dependency Parsing

Cited by in corpus (2)

• Perceived Trustworthiness of Natural Language Generators
• Analysis of the Evolution of Advanced Transformer-Based Language Models: Experiments on Opinion Mining