Neural Networks Compression for Language Modeling

Abstract

In this paper, we consider several compression techniques for the language modeling problem based on recurrent neural networks (RNNs). It is known that conventional RNNs, e.g, LSTM-based networks in language modeling, are characterized with either high space complexity or substantial inference time. This problem is especially crucial for mobile applications, in which the constant interaction with the remote server is inappropriate. By using the Penn Treebank (PTB) dataset we compare pruning, quantization, low-rank factorization, tensor train decomposition for LSTM networks in terms of model size and suitability for fast inference.

Keywords: LSTM, RNN, language modeling, low-rank factorization, pruning, quantization. Published by Springer in the LNCS series, 7th International Conference on Pattern Recognition and Machine Intelligence, 2017

References in corpus (3)

• Recurrent Neural Network Regularization
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• Ultimate tensorization: compressing convolutional and FC layers alike

Cited by in corpus (7)

• Compression of Recurrent Neural Networks for Efficient Language Modeling
• Compressing RNNs for IoT devices by 15-38x using Kronecker Products
• On the Effectiveness of Low-Rank Matrix Factorization for LSTM Model Compression
• Run-Time Efficient RNN Compression for Inference on Edge Devices
• Convolutional neural network compression for natural language processing
• Doping: A technique for efficient compression of LSTM models using sparse structured additive matrices
• A Survey on Green Deep Learning