Regularizing and Optimizing LSTM Language Models

Abstract

Recurrent neural networks (RNNs), such as long short-term memory networks (LSTMs), serve as a fundamental building block for many sequence learning tasks, including machine translation, language modeling, and question answering. In this paper, we consider the specific problem of word-level language modeling and investigate strategies for regularizing and optimizing LSTM-based models. We propose the weight-dropped LSTM which uses DropConnect on hidden-to-hidden weights as a form of recurrent regularization. Further, we introduce NT-ASGD, a variant of the averaged stochastic gradient method, wherein the averaging trigger is determined using a non-monotonic condition as opposed to being tuned by the user. Using these and other regularization strategies, we achieve state-of-the-art word level perplexities on two data sets: 57.3 on Penn Treebank and 65.8 on WikiText-2. In exploring the effectiveness of a neural cache in conjunction with our proposed model, we achieve an even lower state-of-the-art perplexity of 52.8 on Penn Treebank and 52.0 on WikiText-2.

References in corpus (15)

• Adam: A Method for Stochastic Optimization
• Neural Architecture Search with Reinforcement Learning
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Recurrent Neural Network Regularization
• The Marginal Value of Adaptive Gradient Methods in Machine Learning
• Pointer Sentinel Mixture Models
• Train faster, generalize better: Stability of stochastic gradient descent
• Quasi-Recurrent Neural Networks
• Unitary Evolution Recurrent Neural Networks
• Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling
• Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations
• Stochastic Gradient Descent as Approximate Bayesian Inference
• Query-Reduction Networks for Question Answering
• Unbiasing Truncated Backpropagation Through Time
• Revisiting Activation Regularization for Language RNNs

Cited by in corpus (163)

• An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
• fastai: A Layered API for Deep Learning
• CTRL: A Conditional Transformer Language Model for Controllable Generation
• Efficient Neural Architecture Search via Parameter Sharing
• Born Again Neural Networks
• Lookahead Optimizer: k steps forward, 1 step back
• A Gated Recurrent Unit Approach to Bitcoin Price Prediction
• Classification of Hand Movements from EEG using a Deep Attention-based LSTM Network
• Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batches
• Neural Voice Cloning with a Few Samples
• A Simple Baseline for Bayesian Uncertainty in Deep Learning
• Evaluating the Search Phase of Neural Architecture Search
• FinBERT: Financial Sentiment Analysis with Pre-trained Language Models
• Fantastic Generalization Measures and Where to Find Them
• Weighted Transformer Network for Machine Translation
• An Analysis of Neural Language Modeling at Multiple Scales
• FastGRNN: A Fast, Accurate, Stable and Tiny Kilobyte Sized Gated Recurrent Neural Network
• PrivFT: Private and Fast Text Classification with Homomorphic Encryption
• Understanding and Improving Knowledge Distillation
• Multiplicative LSTM for sequence modelling
• ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training
• Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks
• Breaking the Softmax Bottleneck: A High-Rank RNN Language Model
• A Survey of Active Learning for Text Classification using Deep Neural Networks
• Dynamic Evaluation of Neural Sequence Models
• A Survey of Deep Learning for Scientific Discovery
• Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learning
• Insights on representational similarity in neural networks with canonical correlation
• GamePad: A Learning Environment for Theorem Proving
• A Survey on Transfer Learning in Natural Language Processing
• Augment your batch: better training with larger batches
• Survey of Dropout Methods for Deep Neural Networks
• Plan-And-Write: Towards Better Automatic Storytelling
• Language Models with Transformers
• Low Resource Text Classification with ULMFit and Backtranslation
• Optimizing Millions of Hyperparameters by Implicit Differentiation
• Towards Binary-Valued Gates for Robust LSTM Training
• Neural Networks Versus Conventional Filters for Inertial-Sensor-based Attitude Estimation
• ESCORT: Ethereum Smart COntRacTs Vulnerability Detection using Deep Neural Network and Transfer Learning
• A Tutorial on Deep Latent Variable Models of Natural Language
• Evaluating Language Model Finetuning Techniques for Low-resource Languages
• Localization of Fake News Detection via Multitask Transfer Learning
• Addressing Some Limitations of Transformers with Feedback Memory
• Decentralized Deep Learning with Arbitrary Communication Compression
• An Adaptive and Momental Bound Method for Stochastic Learning
• The Implicit and Explicit Regularization Effects of Dropout
• Training DNNs with Hybrid Block Floating Point
• Modeling Vocabulary for Big Code Machine Learning
• You May Not Need Attention
• Model-based Asynchronous Hyperparameter and Neural Architecture Search
• Randomness In Neural Network Training: Characterizing The Impact of Tooling
• Noisin: Unbiased Regularization for Recurrent Neural Networks
• PepCVAE: Semi-Supervised Targeted Design of Antimicrobial Peptide Sequences
• GNN-LM: Language Modeling based on Global Contexts via GNN
• SDM: Sequential Deep Matching Model for Online Large-scale Recommender System
• Evolution of transfer learning in natural language processing
• Understanding language-elicited EEG data by predicting it from a fine-tuned language model
• Selfish Sparse RNN Training
• Efficient Algorithms for Device Placement of DNN Graph Operators
• Improving Perceptual Quality of Drum Transcription with the Expanded Groove MIDI Dataset
• Story Realization: Expanding Plot Events into Sentences
• On Convergence and Generalization of Dropout Training
• IIITT@LT-EDI-EACL2021-Hope Speech Detection: There is always Hope in Transformers
• Neural Polysynthetic Language Modelling
• Restricted Recurrent Neural Networks
• Automated Speech Generation from UN General Assembly Statements: Mapping Risks in AI Generated Texts
• Calibration, Entropy Rates, and Memory in Language Models
• Domain-specific Communication Optimization for Distributed DNN Training
• Pushing the bounds of dropout
• Fix your classifier: the marginal value of training the last weight layer
• Stochastic Optimization with Heavy-Tailed Noise via Accelerated Gradient Clipping
• Creative GANs for generating poems, lyrics, and metaphors
• CPT: Efficient Deep Neural Network Training via Cyclic Precision
• On the comparability of Pre-trained Language Models
• Efficient Neural Architecture Search via Proximal Iterations
• Improved Analysis of Clipping Algorithms for Non-convex Optimization
• Large Scale Legal Text Classification Using Transformer Models
• Knowledge-Augmented Language Model and its Application to Unsupervised Named-Entity Recognition
• Delta-STN: Efficient Bilevel Optimization for Neural Networks using Structured Response Jacobians
• Incremental Natural Language Processing: Challenges, Strategies, and Evaluation
• Transfer Learning for British Sign Language Modelling
• Breaking the Activation Function Bottleneck through Adaptive Parameterization
• Sparseout: Controlling Sparsity in Deep Networks
• Exploring the Vulnerability of Deep Neural Networks: A Study of Parameter Corruption
• Universal Language Model Fine-Tuning with Subword Tokenization for Polish
• Aggregated Momentum: Stability Through Passive Damping
• Composer Style Classification of Piano Sheet Music Images Using Language Model Pretraining
• Learning Over-Parametrized Two-Layer ReLU Neural Networks beyond NTK
• Nondeterminism and Instability in Neural Network Optimization
• Efficient Decoupled Neural Architecture Search by Structure and Operation Sampling
• Language Modeling with Sparse Product of Sememe Experts
• Dissecting Contextual Word Embeddings: Architecture and Representation
• Multi-task Learning with Sample Re-weighting for Machine Reading Comprehension
• Assessing Language Models with Scaling Properties
• Divide and Conquer: A Deep CASA Approach to Talker-independent Monaural Speaker Separation
• Applying a Pre-trained Language Model to Spanish Twitter Humor Prediction
• Adapting GPT, GPT-2 and BERT Language Models for Speech Recognition
• Learning to Compose Topic-Aware Mixture of Experts for Zero-Shot Video Captioning
• Transformer on a Diet
• Towards Non-saturating Recurrent Units for Modelling Long-term Dependencies
• FineText: Text Classification via Attention-based Language Model Fine-tuning
• A Neural Virtual Anchor Synthesizer based on Seq2Seq and GAN Models
• Low-Shot Classification: A Comparison of Classical and Deep Transfer Machine Learning Approaches
• Predefined Sparseness in Recurrent Sequence Models
• Compressing LSTM Networks by Matrix Product Operators
• WeNet: Weighted Networks for Recurrent Network Architecture Search
• Exploiting Unsupervised Pre-training and Automated Feature Engineering for Low-resource Hate Speech Detection in Polish
• Detecting Hostile Posts using Relational Graph Convolutional Network
• Pagsusuri ng RNN-based Transfer Learning Technique sa Low-Resource Language
• Analyzing Customer Feedback for Product Fit Prediction
• AntMan: Sparse Low-Rank Compression to Accelerate RNN inference
• Direct Output Connection for a High-Rank Language Model
• Extrapolation for Large-batch Training in Deep Learning
• Back from the future: bidirectional CTC decoding using future information in speech recognition
• Straight to the Tree: Constituency Parsing with Neural Syntactic Distance
• Complaint Identification in Social Media with Transformer Networks
• Kite: Automatic speech recognition for unmanned aerial vehicles
• Predicting TED Talk Ratings from Language and Prosody
• RIANN -- A Robust Neural Network Outperforms Attitude Estimation Filters
• Dispersed Exponential Family Mixture VAEs for Interpretable Text Generation
• Inflo: News Categorization and Keyphrase Extraction for Implementation in an Aggregation System
• Multi-Zone Unit for Recurrent Neural Networks
• Exploiting Syntactic Structure for Better Language Modeling: A Syntactic Distance Approach
• Analysing Dropout and Compounding Errors in Neural Language Models
• Hierarchical Hidden Markov Jump Processes for Cancer Screening Modeling
• CProp: Adaptive Learning Rate Scaling from Past Gradient Conformity
• Dropping Networks for Transfer Learning
• Better Long-Range Dependency By Bootstrapping A Mutual Information Regularizer
• AutoShrink: A Topology-aware NAS for Discovering Efficient Neural Architecture
• Exploiting Token and Path-based Representations of Code for Identifying Security-Relevant Commits
• Recurrent Neural Network from Adder's Perspective: Carry-lookahead RNN
• REGMAPR - Text Matching Made Easy
• "BNN - BN = ?": Training Binary Neural Networks without Batch Normalization
• Embedding-based Product Retrieval in Taobao Search
• Broccoli: Sprinkling Lightweight Vocabulary Learning into Everyday Information Diets
• Russian Natural Language Generation: Creation of a Language Modelling Dataset and Evaluation with Modern Neural Architectures
• Weakly-Supervised Neural Response Selection from an Ensemble of Task-Specialised Dialogue Agents
• Gradients are Not All You Need
• Input-to-Output Gate to Improve RNN Language Models
• Eliminating Sharp Minima from SGD with Truncated Heavy-tailed Noise
• Mining for Potent Inhibitors through Artificial Intelligence and Physics: A Unified Methodology for Ligand Based and Structure Based Drug Design
• Physics-informed neural networks for solving thermo-mechanics problems of functionally graded material
• IIITT@Dravidian-CodeMix-FIRE2021: Transliterate or translate? Sentiment analysis of code-mixed text in Dravidian languages
• FastTrees: Parallel Latent Tree-Induction for Faster Sequence Encoding
• Structured in Space, Randomized in Time: Leveraging Dropout in RNNs for Efficient Training
• Adaptive Noise Injection: A Structure-Expanding Regularization for RNN
• Paradigm Shift in Language Modeling: Revisiting CNN for Modeling Sanskrit Originated Bengali and Hindi Language
• Regularizing Recurrent Neural Networks via Sequence Mixup
• Leveraging Audio Gestalt to Predict Media Memorability
• Some Limit Properties of Markov Chains Induced by Stochastic Recursive Algorithms
• Token-wise Curriculum Learning for Neural Machine Translation
• Alternating Synthetic and Real Gradients for Neural Language Modeling
• Restricted Recurrent Neural Tensor Networks: Exploiting Word Frequency and Compositionality
• Towards JointUD: Part-of-speech Tagging and Lemmatization using Recurrent Neural Networks
• Mutual Information Constraints for Monte-Carlo Objectives
• RotationOut as a Regularization Method for Neural Network
• SimpleBooks: Long-term dependency book dataset with simplified English vocabulary for word-level language modeling
• Think Again Networks and the Delta Loss
• Meta-Learning a Dynamical Language Model
• Improving Neural Language Models by Segmenting, Attending, and Predicting the Future
• Learning Sparse Structured Ensembles with SG-MCMC and Network Pruning
• Exploiting News Article Structure for Automatic Corpus Generation of Entailment Datasets
• Adaptive Gradient Method with Resilience and Momentum