Improving Network Slimming with Nonconvex Regularization

Abstract

Convolutional neural networks (CNNs) have developed to become powerful models for various computer vision tasks ranging from object detection to semantic segmentation. However, most of the state-of-the-art CNNs cannot be deployed directly on edge devices such as smartphones and drones, which need low latency under limited power and memory bandwidth. One popular, straightforward approach to compressing CNNs is network slimming, which imposes ${\ell }_1$ regularization on the channel-associated scaling factors via the batch normalization layers during training. Network slimming thereby identifies insignificant channels that can be pruned for inference. In this paper, we propose replacing the ${\ell }_1$ penalty with an alternative nonconvex, sparsity-inducing penalty in order to yield a more compressed and/or accurate CNN architecture. We investigate ${\ell }_p(0<p<1)$, transformed ${\ell }_1$ (T${\ell }_1$), minimax concave penalty (MCP), and smoothly clipped absolute deviation (SCAD) due to their recent successes and popularity in solving sparse optimization problems, such as compressed sensing and variable selection. We demonstrate the effectiveness of network slimming with nonconvex penalties on three neural network architectures -- VGG-19, DenseNet-40, and ResNet-164 -- on standard image classification datasets. Based on the numerical experiments, T${\ell }_1$ preserves model accuracy against channel pruning, ${\ell }_{1/2,3/4}$ yield better compressed models with similar accuracies after retraining as ${\ell }_1$, and MCP and SCAD provide more accurate models after retraining with similar compression as ${\ell }_1$. Network slimming with T${\ell }_1$ regularization also outperforms the latest Bayesian modification of network slimming in compressing a CNN architecture in terms of memory storage while preserving its model accuracy after channel pruning.

arXiv commentversion1 published in ISVC'20; version 2: fixed typo; version3 is the extended version and submitted to a journal; version 4: more typos fixed, official version will be on IEEE Access

References in corpus (18)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Nearly unbiased variable selection under minimax concave penalty
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Nonconcave penalized likelihood with a diverging number of parameters
• Ternary Weight Networks
• Trained Ternary Quantization
• Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
• Pruning Filters for Efficient ConvNets
• Compressing Neural Networks with the Hashing Trick
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Group Sparse Regularization for Deep Neural Networks
• Convolutional neural networks with low-rank regularization
• Learning the Number of Neurons in Deep Networks
• The Power of Sparsity in Convolutional Neural Networks
• ProxQuant: Quantized Neural Networks via Proximal Operators
• Trained Rank Pruning for Efficient Deep Neural Networks