Run-Time Efficient RNN Compression for Inference on Edge Devices

Abstract

Recurrent neural networks can be large and compute-intensive, yet many applications that benefit from RNNs run on small devices with very limited compute and storage capabilities while still having run-time constraints. As a result, there is a need for compression techniques that can achieve significant compression without negatively impacting inference run-time and task accuracy. This paper explores a new compressed RNN cell implementation called Hybrid Matrix Decomposition (HMD) that achieves this dual objective. This scheme divides the weight matrix into two parts - an unconstrained upper half and a lower half composed of rank-1 blocks. This results in output features where the upper sub-vector has "richer" features while the lower-sub vector has "constrained features". HMD can compress RNNs by a factor of 2-4x while having a faster run-time than pruning (Zhu &Gupta, 2017) and retaining more model accuracy than matrix factorization (Grachev et al., 2017). We evaluate this technique on 5 benchmarks spanning 3 different applications, illustrating its generality in the domain of edge computing.

arXiv commentPublished at 4th edition of Workshop on Energy Efficient Machine Learning and Cognitive Computing for Embedded Applications at International Symposium of Computer Architecture 2019, Phoenix, Arizona (https://www.emc2-workshop.com/isca-19) colocated with ISCA 2019

References in corpus (22)

• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Recurrent Neural Network Regularization
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition
• Trained Ternary Quantization
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• Deep, Convolutional, and Recurrent Models for Human Activity Recognition using Wearables
• Hello Edge: Keyword Spotting on Microcontrollers
• CirCNN: Accelerating and Compressing Deep Neural Networks Using Block-CirculantWeight Matrices
• Neural Networks with Few Multiplications
• Compressing Recurrent Neural Network with Tensor Train
• Factorization tricks for LSTM networks
• Structured Transforms for Small-Footprint Deep Learning
• Block-Sparse Recurrent Neural Networks
• Alternating Multi-bit Quantization for Recurrent Neural Networks
• Effective Quantization Methods for Recurrent Neural Networks
• An exploration of parameter redundancy in deep networks with circulant projections
• Neural Networks Compression for Language Modeling
• Compressing RNNs for IoT devices by 15-38x using Kronecker Products
• Learning Compact Recurrent Neural Networks
• Measuring scheduling efficiency of RNNs for NLP applications
• Efficient Recurrent Neural Networks using Structured Matrices in FPGAs

Cited by in corpus (5)

• Edge Intelligence: The Confluence of Edge Computing and Artificial Intelligence
• MicroNets: Neural Network Architectures for Deploying TinyML Applications on Commodity Microcontrollers
• Compressing RNNs for IoT devices by 15-38x using Kronecker Products
• Compressing Language Models using Doped Kronecker Products
• High Throughput Matrix-Matrix Multiplication between Asymmetric Bit-Width Operands