Pointer Sentinel Mixture Models

Abstract

Recent neural network sequence models with softmax classifiers have achieved their best language modeling performance only with very large hidden states and large vocabularies. Even then they struggle to predict rare or unseen words even if the context makes the prediction unambiguous. We introduce the pointer sentinel mixture architecture for neural sequence models which has the ability to either reproduce a word from the recent context or produce a word from a standard softmax classifier. Our pointer sentinel-LSTM model achieves state of the art language modeling performance on the Penn Treebank (70.9 perplexity) while using far fewer parameters than a standard softmax LSTM. In order to evaluate how well language models can exploit longer contexts and deal with more realistic vocabularies and larger corpora we also introduce the freely available WikiText corpus.

References in corpus (2)

• Recurrent Neural Network Regularization
• Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks

Cited by in corpus (21)

• Neural Architecture Search with Reinforcement Learning
• Regularizing and Optimizing LSTM Language Models
• Quasi-Recurrent Neural Networks
• Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling
• Data Noising as Smoothing in Neural Network Language Models
• A Neural Knowledge Language Model
• Dynamic Evaluation of Neural Sequence Models
• Challenges in Data-to-Document Generation
• On the State of the Art of Evaluation in Neural Language Models
• Revisiting Activation Regularization for Language RNNs
• Cortical microcircuits as gated-recurrent neural networks
• A Neural Language Model for Dynamically Representing the Meanings of Unknown Words and Entities in a Discourse
• A Flexible Approach to Automated RNN Architecture Generation
• Unbounded cache model for online language modeling with open vocabulary
• Learning to Attend, Copy, and Generate for Session-Based Query Suggestion
• A Convergence Analysis for A Class of Practical Variance-Reduction Stochastic Gradient MCMC
• Context Models for OOV Word Translation in Low-Resource Languages
• Early Improving Recurrent Elastic Highway Network
• Learning Less-Overlapping Representations
• Exchangeable modelling of relational data: checking sparsity, train-test splitting, and sparse exchangeable Poisson matrix factorization
• Input-to-Output Gate to Improve RNN Language Models