On the Compression of Recurrent Neural Networks with an Application to LVCSR acoustic modeling for Embedded Speech Recognition

Abstract

We study the problem of compressing recurrent neural networks (RNNs). In particular, we focus on the compression of RNN acoustic models, which are motivated by the goal of building compact and accurate speech recognition systems which can be run efficiently on mobile devices. In this work, we present a technique for general recurrent model compression that jointly compresses both recurrent and non-recurrent inter-layer weight matrices. We find that the proposed technique allows us to reduce the size of our Long Short-Term Memory (LSTM) acoustic model to a third of its original size with negligible loss in accuracy.

arXiv commentAccepted in ICASSP 2016

References in corpus (6)

• Distilling the Knowledge in a Neural Network
• Sequence to Sequence Learning with Neural Networks
• Do Deep Nets Really Need to be Deep?
• Predicting Parameters in Deep Learning
• Compressing Neural Networks with the Hashing Trick
• Personalized Speech recognition on mobile devices

Cited by in corpus (16)

• Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces
• Deep Learning in Mobile and Wireless Networking: A Survey
• Learning Intrinsic Sparse Structures within Long Short-Term Memory
• Sequence-Level Knowledge Distillation
• Compression of Recurrent Neural Networks for Efficient Language Modeling
• Learning Compact Recurrent Neural Networks with Block-Term Tensor Decomposition
• Restricted Recurrent Neural Networks
• RNN-T Models Fail to Generalize to Out-of-Domain Audio: Causes and Solutions
• The Incredible Shrinking Neural Network: New Perspectives on Learning Representations Through The Lens of Pruning
• Compression of Acoustic Event Detection Models with Low-rank Matrix Factorization and Quantization Training
• Streaming Small-Footprint Keyword Spotting using Sequence-to-Sequence Models
• Recent Progresses in Deep Learning based Acoustic Models (Updated)
• MobiVSR: A Visual Speech Recognition Solution for Mobile Devices
• In-training Matrix Factorization for Parameter-frugal Neural Machine Translation
• An Embedded Deep Learning based Word Prediction
• Dynamic Encoder Transducer: A Flexible Solution For Trading Off Accuracy For Latency