Pointer Sentinel Mixture Models

Abstract

Recent neural network sequence models with softmax classifiers have achieved their best language modeling performance only with very large hidden states and large vocabularies. Even then they struggle to predict rare or unseen words even if the context makes the prediction unambiguous. We introduce the pointer sentinel mixture architecture for neural sequence models which has the ability to either reproduce a word from the recent context or produce a word from a standard softmax classifier. Our pointer sentinel-LSTM model achieves state of the art language modeling performance on the Penn Treebank (70.9 perplexity) while using far fewer parameters than a standard softmax LSTM. In order to evaluate how well language models can exploit longer contexts and deal with more realistic vocabularies and larger corpora we also introduce the freely available WikiText corpus.

References in corpus (7)

• Recurrent Neural Network Regularization
• A Theoretically Grounded Application of Dropout in Recurrent Neural Networks
• Dynamic Memory Networks for Visual and Textual Question Answering
• Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks
• How to Construct Deep Recurrent Neural Networks
• Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations
• Latent Predictor Networks for Code Generation

Cited by in corpus (247)

• An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
• Neural Architecture Search with Reinforcement Learning
• A Deep Reinforced Model for Abstractive Summarization
• Linformer: Self-Attention with Linear Complexity
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• Dynamic Coattention Networks For Question Answering
• Regularizing and Optimizing LSTM Language Models
• DeBERTa: Decoding-enhanced BERT with Disentangled Attention
• Asynchronous Federated Optimization
• SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning
• Quasi-Recurrent Neural Networks
• R-Drop: Regularized Dropout for Neural Networks
• Reducing Transformer Depth on Demand with Structured Dropout
• Neural Text Generation with Unlikelihood Training
• CLEAR: Contrastive Learning for Sentence Representation
• A Survey of Privacy Attacks in Machine Learning
• On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines
• HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients
• ST-GRAT: A Novel Spatio-temporal Graph Attention Network for Accurately Forecasting Dynamically Changing Road Speed
• Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling
• Data Noising as Smoothing in Neural Network Language Models
• Improving Neural Network Quantization without Retraining using Outlier Channel Splitting
• Fault Detection and Identification using Bayesian Recurrent Neural Networks
• BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model
• Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code
• Bayesian Recurrent Neural Networks
• Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling
• Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN
• Rigging the Lottery: Making All Tickets Winners
• Training with Quantization Noise for Extreme Model Compression
• A Neural Knowledge Language Model
• On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behavior
• Get To The Point: Summarization with Pointer-Generator Networks
• Analyzing Information Leakage of Updates to Natural Language Models
• Multiplicative LSTM for sequence modelling
• Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks
• AdamP: Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weights
• Dynamic Sampling and Selective Masking for Communication-Efficient Federated Learning
• Language GANs Falling Short
• Hopfield Networks is All You Need
• Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks
• Backdoor Pre-trained Models Can Transfer to All
• Gender Bias in Neural Natural Language Processing
• Full Page Handwriting Recognition via Image to Sequence Extraction
• Breaking the Softmax Bottleneck: A High-Rank RNN Language Model
• Dynamic Evaluation of Neural Sequence Models
• Generalization through Memorization: Nearest Neighbor Language Models
• Challenges in Data-to-Document Generation
• Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioning
• A Constructive Prediction of the Generalization Error Across Scales
• On the State of the Art of Evaluation in Neural Language Models
• Compressive Transformers for Long-Range Sequence Modelling
• Hash Layers For Large Sparse Models
• Tensorized Embedding Layers for Efficient Model Compression
• Asynchronous Federated Learning on Heterogeneous Devices: A Survey
• Language Models with Transformers
• Knowledge-Aware Language Model Pretraining
• Low Resource Text Classification with ULMFit and Backtranslation
• Revisiting Activation Regularization for Language RNNs
• On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent
• Mixed-Precision Training for NLP and Speech Recognition with OpenSeq2Seq
• Towards Binary-Valued Gates for Robust LSTM Training
• Evaluating prose style transfer with the Bible
• RedSync : Reducing Synchronization Traffic for Distributed Deep Learning
• Temporal Convolutional Attention-based Network For Sequence Modeling
• Cortical microcircuits as gated-recurrent neural networks
• Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
• Decentralized Deep Learning with Arbitrary Communication Compression
• Real or Fake? Learning to Discriminate Machine from Human Generated Text
• Improving the Gating Mechanism of Recurrent Neural Networks
• Machine Reading Comprehension: a Literature Review
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• The Implicit and Explicit Regularization Effects of Dropout
• Relational recurrent neural networks
• Scaling Laws for Deep Learning
• A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
• Modeling Vocabulary for Big Code Machine Learning
• Learning to Encode Position for Transformer with Continuous Dynamical Model
• Reducing Sentiment Bias in Language Models via Counterfactual Evaluation
• Attention-based Modeling for Emotion Detection and Classification in Textual Conversations
• Deriving Neural Architectures from Sequence and Graph Kernels
• Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability
• Neural Language Modeling by Jointly Learning Syntax and Lexicon
• Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition
• Fast Parametric Learning with Activation Memorization
• Adaptively Truncating Backpropagation Through Time to Control Gradient Bias
• A Programmable Approach to Neural Network Compression
• Sigsoftmax: Reanalysis of the Softmax Bottleneck
• Self-Knowledge Distillation in Natural Language Processing
• Noisin: Unbiased Regularization for Recurrent Neural Networks
• Implicit Dimension Identification in User-Generated Text with LSTM Networks
• On Generalization Bounds of a Family of Recurrent Neural Networks
• A Transfer-Learnable Natural Language Interface for Databases
• GNN-LM: Language Modeling based on Global Contexts via GNN
• Zero-Shot Dialog Generation with Cross-Domain Latent Actions
• Selfish Sparse RNN Training
• Production Ready Chatbots: Generate if not Retrieve
• Understanding language-elicited EEG data by predicting it from a fine-tuned language model
• Can I trust you more? Model-Agnostic Hierarchical Explanations
• Financial Aspect-Based Sentiment Analysis using Deep Representations
• Curriculum learning for language modeling
• Restricted Recurrent Neural Networks
• Communication-Efficient Federated Distillation
• A Neural Language Model for Dynamically Representing the Meanings of Unknown Words and Entities in a Discourse
• Split and Rephrase: Better Evaluation and a Stronger Baseline
• Domain-specific Communication Optimization for Distributed DNN Training
• Fix your classifier: the marginal value of training the last weight layer
• Ranking and Tuning Pre-trained Models: A New Paradigm for Exploiting Model Hubs
• A Flexible Approach to Automated RNN Architecture Generation
• Unbounded cache model for online language modeling with open vocabulary
• Differentiable Architecture Search with Ensemble Gumbel-Softmax
• AC/DC: Alternating Compressed/DeCompressed Training of Deep Neural Networks
• CPT: Efficient Deep Neural Network Training via Cyclic Precision
• On the comparability of Pre-trained Language Models
• Generative Models for Simulating Mobility Trajectories
• Reference-Aware Language Models
• AMR Parsing as Sequence-to-Graph Transduction
• Understanding Recurrent Neural Architectures by Analyzing and Synthesizing Long Distance Dependencies in Benchmark Sequential Datasets
• Knowledge-Augmented Language Model and its Application to Unsupervised Named-Entity Recognition
• Scatterbrain: Unifying Sparse and Low-rank Attention Approximation
• LightPAFF: A Two-Stage Distillation Framework for Pre-training and Fine-tuning
• Improving Interpretability of Word Embeddings by Generating Definition and Usage
• Revisiting BFloat16 Training
• Breaking the Activation Function Bottleneck through Adaptive Parameterization
• Adversarial Training with Contrastive Learning in NLP
• Statistical Adaptive Stochastic Gradient Methods
• Self-Normalization Properties of Language Modeling
• Learning to Summarize Radiology Findings
• Multi-objective Asynchronous Successive Halving
• Consistency of a Recurrent Language Model With Respect to Incomplete Decoding
• A Convergence Analysis for A Class of Practical Variance-Reduction Stochastic Gradient MCMC
• DQI: Measuring Data Quality in NLP
• Sparseout: Controlling Sparsity in Deep Networks
• Term Revealing: Furthering Quantization at Run Time on Quantized DNNs
• Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models
• FTRANS: Energy-Efficient Acceleration of Transformers using FPGA
• Learning to Attend, Copy, and Generate for Session-Based Query Suggestion
• A Comparison of Approaches to Document-level Machine Translation
• MultiFiT: Efficient Multi-lingual Language Model Fine-tuning
• Summarize, Outline, and Elaborate: Long-Text Generation via Hierarchical Supervision from Extractive Summaries
• LazyTensor: combining eager execution with domain-specific compilers
• Compressing Deep Neural Networks via Layer Fusion
• Extension of Direct Feedback Alignment to Convolutional and Recurrent Neural Network for Bio-plausible Deep Learning
• AdaSGD: Bridging the gap between SGD and Adam
• Communication-Efficient Federated Learning with Compensated Overlap-FedAvg
• Hyperparameter Optimization Is Deceiving Us, and How to Stop It
• Not All Memories are Created Equal: Learning to Forget by Expiring
• Compressing Gradient Optimizers via Count-Sketches
• Pay Attention when Required
• Pufferfish: Communication-efficient Models At No Extra Cost
• Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures
• Progress and Tradeoffs in Neural Language Models
• Parameter Re-Initialization through Cyclical Batch Size Schedules
• Assessing Language Models with Scaling Properties
• Offensive Language Identification in Low-resourced Code-mixed Dravidian languages using Pseudo-labeling
• Transformer on a Diet
• Early Improving Recurrent Elastic Highway Network
• FineText: Text Classification via Attention-based Language Model Fine-tuning
• Distributed Deep Learning in Open Collaborations
• Adding Recurrence to Pretrained Transformers for Improved Efficiency and Context Size
• Context Models for OOV Word Translation in Low-Resource Languages
• A Generalizable Approach to Learning Optimizers
• Pun Generation with Surprise
• Adaptive Pruning of Neural Language Models for Mobile Devices
• SLSGD: Secure and Efficient Distributed On-device Machine Learning
• Be Concise and Precise: Synthesizing Open-Domain Entity Descriptions from Facts
• Discovering Useful Sentence Representations from Large Pretrained Language Models
• Block-wise Dynamic Sparseness
• Microsoft AI Challenge India 2018: Learning to Rank Passages for Web Question Answering with Deep Attention Networks
• Variational Smoothing in Recurrent Neural Network Language Models
• Black-Box Attacks on Sequential Recommenders via Data-Free Model Extraction
• Learning Less-Overlapping Representations
• Non-entailed subsequences as a challenge for natural language inference
• Direct Output Connection for a High-Rank Language Model
• Word Embedding based on Low-Rank Doubly Stochastic Matrix Decomposition
• Multi-scale Transformer Language Models
• Is Attention All What You Need? -- An Empirical Investigation on Convolution-Based Active Memory and Self-Attention
• Doubly Sparse: Sparse Mixture of Sparse Experts for Efficient Softmax Inference
• Learning to Adaptively Scale Recurrent Neural Networks
• Character n-gram Embeddings to Improve RNN Language Models
• Linguistically Informed Masking for Representation Learning in the Patent Domain
• Dual Averaging is Surprisingly Effective for Deep Learning Optimization
• Language Model Evaluation in Open-ended Text Generation
• Limitations of Autoregressive Models and Their Alternatives
• On Learning the Transformer Kernel
• Adversarial Black-Box Attacks On Text Classifiers Using Multi-Objective Genetic Optimization Guided By Deep Networks
• A Mixture of $h-1$ Heads is Better than $h$ Heads
• MLE-guided parameter search for task loss minimization in neural sequence modeling
• Extrapolation for Large-batch Training in Deep Learning
• WikiCheck: An end-to-end open source Automatic Fact-Checking API based on Wikipedia
• Long Short-Term Memory with Dynamic Skip Connections
• End-to-End Argument Mining for Discussion Threads Based on Parallel Constrained Pointer Architecture
• Multi-Source Pointer Network for Product Title Summarization
• Do Transformers Need Deep Long-Range Memory
• Distributional Discrepancy: A Metric for Unconditional Text Generation
• Exploiting Syntactic Structure for Better Language Modeling: A Syntactic Distance Approach
• Russian Natural Language Generation: Creation of a Language Modelling Dataset and Evaluation with Modern Neural Architectures
• Word Interdependence Exposes How LSTMs Compose Representations
• Conditional Generators of Words Definitions
• Broccoli: Sprinkling Lightweight Vocabulary Learning into Everyday Information Diets
• Solon: Communication-efficient Byzantine-resilient Distributed Training via Redundant Gradients
• Compositional Attention: Disentangling Search and Retrieval
• Analysing Dropout and Compounding Errors in Neural Language Models
• Assessing the Reliability of Word Embedding Gender Bias Measures
• Exploring the Use of Attention within an Neural Machine Translation Decoder States to Translate Idioms
• More Than Words: Collocation Tokenization for Latent Dirichlet Allocation Models
• Phrase-BERT: Improved Phrase Embeddings from BERT with an Application to Corpus Exploration
• Plan ahead: Self-Supervised Text Planning for Paragraph Completion Task
• Random Network Distillation as a Diversity Metric for Both Image and Text Generation
• Memory Augmented Optimizers for Deep Learning
• Amanuensis: The Programmer's Apprentice
• Approximate Fixed-Points in Recurrent Neural Networks
• A Lightweight Recurrent Network for Sequence Modeling
• Efficient Estimation of Influence of a Training Instance
• Adversarial Decomposition of Text Representation
• Better Long-Range Dependency By Bootstrapping A Mutual Information Regularizer
• Cascaded Head-colliding Attention
• Topic, Sentiment and Impact Analysis: COVID19 Information Seeking on Social Media
• Sparse Meta Networks for Sequential Adaptation and its Application to Adaptive Language Modelling
• Mode recovery in neural autoregressive sequence modeling
• Recognising Biomedical Names: Challenges and Solutions
• Frequency-aware SGD for Efficient Embedding Learning with Provable Benefits
• Capturing Structural Locality in Non-parametric Language Models
• Exchangeable modelling of relational data: checking sparsity, train-test splitting, and sparse exchangeable Poisson matrix factorization
• Benchmarking down-scaled (not so large) pre-trained language models
• Dispatcher: A Message-Passing Approach To Language Modelling
• Revisiting Neural Language Modelling with Syllables
• Cross Copy Network for Dialogue Generation
• LSTMs Compose (and Learn) Bottom-Up
• Anomaly-Injected Deep Support Vector Data Description for Text Outlier Detection
• Towards Zero-Shot Knowledge Distillation for Natural Language Processing
• Token-wise Curriculum Learning for Neural Machine Translation
• MicroNet for Efficient Language Modeling
• Ouroboros: On Accelerating Training of Transformer-Based Language Models
• TransSent: Towards Generation of Structured Sentences with Discourse Marker
• A Random Gossip BMUF Process for Neural Language Modeling
• Adaptive Noise Injection: A Structure-Expanding Regularization for RNN
• Alternating Synthetic and Real Gradients for Neural Language Modeling
• Classification as Decoder: Trading Flexibility for Control in Medical Dialogue
• SimpleBooks: Long-term dependency book dataset with simplified English vocabulary for word-level language modeling
• Low-Complexity LSTM Training and Inference with FloatSD8 Weight Representation
• Meta-Learning a Dynamical Language Model
• Where's My Head? Definition, Dataset and Models for Numeric Fused-Heads Identification and Resolution
• OrderNet: Ordering by Example
• Adaptive Recurrent Neural Network Based on Mixture Layer
• Improving Neural Language Models by Segmenting, Attending, and Predicting the Future
• Input-to-Output Gate to Improve RNN Language Models