Group Pruning using a Bounded-Lp norm for Group Gating and Regularization

Abstract

Deep neural networks achieve state-of-the-art results on several tasks while increasing in complexity. It has been shown that neural networks can be pruned during training by imposing sparsity inducing regularizers. In this paper, we investigate two techniques for group-wise pruning during training in order to improve network efficiency. We propose a gating factor after every convolutional layer to induce channel level sparsity, encouraging insignificant channels to become exactly zero. Further, we introduce and analyse a bounded variant of the L1 regularizer, which interpolates between L1 and L0-norms to retain performance of the network at higher pruning rates. To underline effectiveness of the proposed methods,we show that the number of parameters of ResNet-164, DenseNet-40 and MobileNetV2 can be reduced down by 30%, 69% and 75% on CIFAR100 respectively without a significant drop in accuracy. We achieve state-of-the-art pruning results for ResNet-50 with higher accuracy on ImageNet. Furthermore, we show that the light weight MobileNetV2 can further be compressed on ImageNet without a significant drop in performance.

arXiv commentGerman Conference on Pattern Recognition (GCPR) 2019, 12 main pages, 3 pages of appendix, 4 figures, 2 tables

References in corpus (23)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Compressing Deep Convolutional Networks using Vector Quantization
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
• Compressing Neural Networks with the Hashing Trick
• Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
• Bayesian Compression for Deep Learning
• Variational Dropout Sparsifies Deep Neural Networks
• Training and Inference with Integers in Deep Neural Networks
• Learning the Number of Neurons in Deep Networks
• Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers
• Structured Bayesian Pruning via Log-Normal Multiplicative Noise
• Soft Weight-Sharing for Neural Network Compression
• Efficient Processing of Deep Neural Networks: A Tutorial and Survey
• Data-Driven Sparse Structure Selection for Deep Neural Networks
• Structured Variational Learning of Bayesian Neural Networks with Horseshoe Priors
• Improved Bayesian Compression
• Automatic Relevance Determination For Deep Generative Models