On the Compression of Neural Networks Using ${\ell }_0$-Norm Regularization and Weight Pruning

Abstract

Despite the growing availability of high-capacity computational platforms, implementation complexity still has been a great concern for the real-world deployment of neural networks. This concern is not exclusively due to the huge costs of state-of-the-art network architectures, but also due to the recent push towards edge intelligence and the use of neural networks in embedded applications. In this context, network compression techniques have been gaining interest due to their ability for reducing deployment costs while keeping inference accuracy at satisfactory levels. The present paper is dedicated to the development of a novel compression scheme for neural networks. To this end, a new form of ${\ell }_0$-norm-based regularization is firstly developed, which is capable of inducing strong sparseness in the network during training. Then, targeting the smaller weights of the trained network with pruning techniques, smaller yet highly effective networks can be obtained. The proposed compression scheme also involves the use of ${\ell }_2$-norm regularization to avoid overfitting as well as fine tuning to improve the performance of the pruned network. Experimental results are presented aiming to show the effectiveness of the proposed scheme as well as to make comparisons with competing approaches.

arXiv comment34 pages, 9 figures, 7 tables

References in corpus (10)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Group Sparse Regularization for Deep Neural Networks
• l_0 Norm Constraint LMS Algorithm for Sparse System Identification
• Bayesian Compression for Deep Learning
• Variational Dropout Sparsifies Deep Neural Networks
• Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers
• Global Sparse Momentum SGD for Pruning Very Deep Neural Networks
• DropNeuron: Simplifying the Structure of Deep Neural Networks
• $L_0$-ARM: Network Sparsification via Stochastic Binary Optimization