A Survey on Deep Neural Network Compression: Challenges, Overview, and Solutions

Abstract

Deep Neural Network (DNN) has gained unprecedented performance due to its automated feature extraction capability. This high order performance leads to significant incorporation of DNN models in different Internet of Things (IoT) applications in the past decade. However, the colossal requirement of computation, energy, and storage of DNN models make their deployment prohibitive on resource constraint IoT devices. Therefore, several compression techniques were proposed in recent years for reducing the storage and computation requirements of the DNN model. These techniques on DNN compression have utilized a different perspective for compressing DNN with minimal accuracy compromise. It encourages us to make a comprehensive overview of the DNN compression techniques. In this paper, we present a comprehensive review of existing literature on compressing DNN model that reduces both storage and computation requirements. We divide the existing approaches into five broad categories, i.e., network pruning, sparse representation, bits precision, knowledge distillation, and miscellaneous, based upon the mechanism incorporated for compressing the DNN model. The paper also discussed the challenges associated with each category of DNN compression techniques. Finally, we provide a quick summary of existing work under each category with the future direction in DNN compression.

arXiv comment19 pages, 9 figures

References in corpus (12)

• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• A survey of sparse representation: algorithms and applications
• Binarized Neural Networks
• Bayesian Compression for Deep Learning
• Towards Understanding Knowledge Distillation
• Sparse DNNs with Improved Adversarial Robustness
• What is the State of Neural Network Pruning?
• Multi-Task Zipping via Layer-wise Neuron Sharing
• ShrinkTeaNet: Million-scale Lightweight Face Recognition via Shrinking Teacher-Student Networks

Cited by in corpus (7)

• Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better
• Compacting Deep Neural Networks for Internet of Things: Methods and Applications
• MD Loss: Efficient Training of 3D Seismic Fault Segmentation Network under Sparse Labels by Weakening Anomaly Annotation
• Principled Pruning of Bayesian Neural Networks through Variational Free Energy Minimization
• Model Fusion via Neuron Transplantation
• CHISEL: Compression-Aware High-Accuracy Embedded Indoor Localization with Deep Learning
• Performance optimizations on deep noise suppression models