Doping: A technique for efficient compression of LSTM models using sparse structured additive matrices

Abstract

Structured matrices, such as those derived from Kronecker products (KP), are effective at compressing neural networks, but can lead to unacceptable accuracy loss when applied to large models. In this paper, we propose the notion of doping -- addition of an extremely sparse matrix to a structured matrix. Doping facilitates additional degrees of freedom for a small number of parameters, allowing them to independently diverge from the fixed structure. To train LSTMs with doped structured matrices, we introduce the additional parameter matrix while slowly annealing its sparsity level. However, we find that performance degrades as we slowly sparsify the doping matrix, due to co-matrix adaptation (CMA) between the structured and the sparse matrices. We address this over dependence on the sparse matrix using a co-matrix dropout regularization (CMR) scheme. We provide empirical evidence to show that doping, CMA and CMR are concepts generally applicable to multiple structured matrices (Kronecker Product, LMF, Hybrid Matrix Decomposition). Additionally, results with doped kronecker product matrices demonstrate state-of-the-art accuracy at large compression factors (10 - 25x) across 4 natural language processing applications with minor loss in accuracy. Doped KP compression technique outperforms previous state-of-the art compression results by achieving 1.3 - 2.4x higher compression factor at a similar accuracy, while also beating strong alternatives like pruning and low-rank methods by a large margin (8% or more). Additionally, we show that doped KP can be deployed on commodity hardware using the current software stack and achieve 2.5 - 5.5x inference run-time speed-up over baseline.

arXiv commentAccepted to be published at MLSys 2021

References in corpus (20)

• Deep Residual Learning for Image Recognition
• Recurrent Neural Network Regularization
• Binarized Neural Networks
• The State of Sparsity in Deep Neural Networks
• MicroNets: Neural Network Architectures for Deploying TinyML Applications on Commodity Microcontrollers
• Structured Bayesian Pruning via Log-Normal Multiplicative Noise
• TinyLSTMs: Efficient Neural Speech Enhancement for Hearing Aids
• Compressing Recurrent Neural Network with Tensor Train
• Factorization tricks for LSTM networks
• Structured Transforms for Small-Footprint Deep Learning
• Compression of Recurrent Neural Networks for Efficient Language Modeling
• FixyNN: Efficient Hardware for Mobile Computer Vision via Transfer Learning
• Movement Pruning: Adaptive Sparsity by Fine-Tuning
• Neural Networks Compression for Language Modeling
• Compressing RNNs for IoT devices by 15-38x using Kronecker Products
• Exploiting Local Structures with the Kronecker Layer in Convolutional Networks
• Splitting Steepest Descent for Growing Neural Architectures
• Mobile Machine Learning Hardware at ARM: A Systems-on-Chip (SoC) Perspective
• Retraining-Based Iterative Weight Quantization for Deep Neural Networks
• Pyramidal Recurrent Unit for Language Modeling

Cited by in corpus (1)

• MicroNets: Neural Network Architectures for Deploying TinyML Applications on Commodity Microcontrollers