LightRNN: Memory and Computation-Efficient Recurrent Neural Networks

Abstract

Recurrent neural networks (RNNs) have achieved state-of-the-art performances in many natural language processing tasks, such as language modeling and machine translation. However, when the vocabulary is large, the RNN model will become very big (e.g., possibly beyond the memory capacity of a GPU device) and its training will become very inefficient. In this work, we propose a novel technique to tackle this challenge. The key idea is to use 2-Component (2C) shared embedding for word representations. We allocate every word in the vocabulary into a table, each row of which is associated with a vector, and each column associated with another vector. Depending on its position in the table, a word is jointly represented by two components: a row vector and a column vector. Since the words in the same row share the row vector and the words in the same column share the column vector, we only need $2\sqrt{|V|}$ vectors to represent a vocabulary of $|V|$ unique words, which are far less than the $|V|$ vectors required by existing approaches. Based on the 2-Component shared embedding, we design a new RNN algorithm and evaluate it using the language modeling task on several benchmark datasets. The results show that our algorithm significantly reduces the model size and speeds up the training process, without sacrifice of accuracy (it achieves similar, if not better, perplexity as compared to state-of-the-art language models). Remarkably, on the One-Billion-Word benchmark Dataset, our algorithm achieves comparable perplexity to previous language models, whilst reducing the model size by a factor of 40-100, and speeding up the training process by a factor of 2. We name our proposed algorithm \emph{LightRNN} to reflect its very small model size and very high training speed.

NIPS 2016

References in corpus (7)

• Sequence to Sequence Learning with Neural Networks
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Recurrent Neural Network Regularization
• Character-Aware Neural Language Models
• Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognition
• Compositional Morphology for Word Representations and Language Modelling
• Optimizing Performance of Recurrent Neural Networks on GPUs

Cited by in corpus (10)

• On-Device Machine Learning: An Algorithms and Learning Theory Perspective
• An Interpretable Knowledge Transfer Model for Knowledge Base Completion
• Fast Variational AutoEncoder with Inverted Multi-Index for Collaborative Filtering
• Slim Embedding Layers for Recurrent Neural Language Models
• Learning K-way D-dimensional Discrete Code For Compact Embedding Representations
• Character-Level Language Modeling with Hierarchical Recurrent Neural Networks
• Disentangled Item Representation for Recommender Systems
• Physics-Guided Generative Adversarial Networks for Sea Subsurface Temperature Prediction
• Image Captioning with Sparse Recurrent Neural Network
• Fast and Simple Mixture of Softmaxes with BPE and Hybrid-LightRNN for Language Generation