Compressing RNNs for IoT devices by 15-38x using Kronecker Products

Abstract

Recurrent Neural Networks (RNN) can be difficult to deploy on resource constrained devices due to their size.As a result, there is a need for compression techniques that can significantly compress RNNs without negatively impacting task accuracy. This paper introduces a method to compress RNNs for resource constrained environments using Kronecker product (KP). KPs can compress RNN layers by 15-38x with minimal accuracy loss. By quantizing the resulting models to 8-bits, we further push the compression factor to 50x. We show that KP can beat the task accuracy achieved by other state-of-the-art compression techniques across 5 benchmarks spanning 3 different applications, while simultaneously improving inference run-time. We show that the KP compression mechanism does introduce an accuracy loss, which can be mitigated by a proposed hybrid KP (HKP) approach. Our HKP algorithm provides fine-grained control over the compression ratio, enabling us to regain accuracy lost during compression by adding a small number of model parameters.

References in corpus (29)

• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition
• Trained Ternary Quantization
• Predicting Parameters in Deep Learning
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• The State of Sparsity in Deep Neural Networks
• Deep, Convolutional, and Recurrent Models for Human Activity Recognition using Wearables
• Hello Edge: Keyword Spotting on Microcontrollers
• Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition
• Unitary Evolution Recurrent Neural Networks
• Neural Networks with Few Multiplications
• Structured Bayesian Pruning via Log-Normal Multiplicative Noise
• Compressing Recurrent Neural Network with Tensor Train
• Structured Transforms for Small-Footprint Deep Learning
• Factorization tricks for LSTM networks
• FastGRNN: A Fast, Accurate, Stable and Tiny Kilobyte Sized Gated Recurrent Neural Network
• Block-Sparse Recurrent Neural Networks
• The Power of Sparsity in Convolutional Neural Networks
• Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNs
• Effective Quantization Methods for Recurrent Neural Networks
• The Unreasonable Effectiveness of Structured Random Orthogonal Embeddings
• Neural Networks Compression for Language Modeling
• Run-Time Efficient RNN Compression for Inference on Edge Devices
• Measuring scheduling efficiency of RNNs for NLP applications
• Structured Weight Matrices-Based Hardware Accelerators in Deep Neural Networks: FPGAs and ASICs
• Kronecker Recurrent Units
• Compression of Fully-Connected Layer in Neural Network by Kronecker Product
• Learning Compressed Transforms with Low Displacement Rank

Cited by in corpus (8)

• MicroNets: Neural Network Architectures for Deploying TinyML Applications on Commodity Microcontrollers
• Compacting Deep Neural Networks for Internet of Things: Methods and Applications
• Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers
• Run-Time Efficient RNN Compression for Inference on Edge Devices
• Compressing Language Models using Doped Kronecker Products
• High Throughput Matrix-Matrix Multiplication between Asymmetric Bit-Width Operands
• DARB: A Density-Aware Regular-Block Pruning for Deep Neural Networks
• Doping: A technique for efficient compression of LSTM models using sparse structured additive matrices