Compressing Neural Language Models by Sparse Word Representations

Abstract

Neural networks are among the state-of-the-art techniques for language modeling. Existing neural language models typically map discrete words to distributed, dense vector representations. After information processing of the preceding context words by hidden layers, an output layer estimates the probability of the next word. Such approaches are time- and memory-intensive because of the large numbers of parameters for word embeddings and the output layer. In this paper, we propose to compress neural language models by sparse word representations. In the experiments, the number of parameters in our model increases very slowly with the growth of the vocabulary size, which is almost imperceptible. Moreover, our approach not only reduces the parameter space to a large extent, but also improves the performance in terms of the perplexity measure.

ACL-16, pages 226--235

References in corpus (7)

• Adam: A Method for Stochastic Optimization
• Distilling the Knowledge in a Neural Network
• Compressing Deep Convolutional Networks using Vector Quantization
• Visualizing and Understanding Recurrent Networks
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Strategies for Training Large Vocabulary Neural Language Models
• Distilling Word Embeddings: An Encoding Approach

Cited by in corpus (5)

• DeFINE: DEep Factorized INput Token Embeddings for Neural Sequence Modeling
• Anchor & Transform: Learning Sparse Embeddings for Large Vocabularies
• Sparse Coding of Neural Word Embeddings for Multilingual Sequence Labeling
• A Survey on Green Deep Learning
• Sparsity Emerges Naturally in Neural Language Models