Breaking the Softmax Bottleneck: A High-Rank RNN Language Model

Abstract

We formulate language modeling as a matrix factorization problem, and show that the expressiveness of Softmax-based models (including the majority of neural language models) is limited by a Softmax bottleneck. Given that natural language is highly context-dependent, this further implies that in practice Softmax with distributed word embeddings does not have enough capacity to model natural language. We propose a simple and effective method to address this issue, and improve the state-of-the-art perplexities on Penn Treebank and WikiText-2 to 47.69 and 40.68 respectively. The proposed method also excels on the large-scale 1B Word dataset, outperforming the baseline by over 5.6 points in perplexity.

arXiv commentICLR Oral 2018

References in corpus (14)

• Neural Architecture Search with Reinforcement Learning
• Recurrent Neural Network Regularization
• Character-Aware Neural Language Models
• DRAW: A Recurrent Neural Network For Image Generation
• Pointer Sentinel Mixture Models
• Regularizing and Optimizing LSTM Language Models
• Hierarchical Multiscale Recurrent Neural Networks
• Learning Stochastic Recurrent Networks
• Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling
• Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders
• Sequential Neural Models with Stochastic Layers
• Learning Factored Representations in a Deep Mixture of Experts
• Recurrent Mixture Density Network for Spatiotemporal Visual Attention
• Deep Temporal Sigmoid Belief Networks for Sequence Modeling

Cited by in corpus (26)

• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• Evaluating the Search Phase of Neural Architecture Search
• Robust sound event detection in bioacoustic sensor networks
• Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks
• Tensorized Embedding Layers for Efficient Model Compression
• Relational recurrent neural networks
• Fast Parametric Learning with Activation Memorization
• Noisin: Unbiased Regularization for Recurrent Neural Networks
• Stochastic WaveNet: A Generative Latent Variable Model for Sequential Data
• Deep Complex Networks for Protocol-Agnostic Radio Frequency Device Fingerprinting in the Wild
• Low-Rank Bottleneck in Multi-head Attention Models
• Pushing the bounds of dropout
• Universal Language Model Fine-Tuning with Subword Tokenization for Polish
• Efficient Decoupled Neural Architecture Search by Structure and Operation Sampling
• Improving Pre-Trained Multilingual Models with Vocabulary Expansion
• Adaptive Pruning of Neural Language Models for Mobile Devices
• Variational Smoothing in Recurrent Neural Network Language Models
• Doubly Sparse: Sparse Mixture of Sparse Experts for Efficient Softmax Inference
• Better Long-Range Dependency By Bootstrapping A Mutual Information Regularizer
• Knowledge Efficient Deep Learning for Natural Language Processing
• Graphmax for Text Generation
• Learning Sparse Structured Ensembles with SG-MCMC and Network Pruning
• Adaptive Noise Injection: A Structure-Expanding Regularization for RNN
• Error-Correcting Neural Sequence Prediction
• Quantum Statistics-Inspired Neural Attention
• Improving Neural Language Models by Segmenting, Attending, and Predicting the Future