OICSR: Out-In-Channel Sparsity Regularization for Compact Deep Neural Networks

Abstract

Channel pruning can significantly accelerate and compress deep neural networks. Many channel pruning works utilize structured sparsity regularization to zero out all the weights in some channels and automatically obtain structure-sparse network in training stage. However, these methods apply structured sparsity regularization on each layer separately where the correlations between consecutive layers are omitted. In this paper, we first combine one out-channel in current layer and the corresponding in-channel in next layer as a regularization group, namely out-in-channel. Our proposed Out-In-Channel Sparsity Regularization (OICSR) considers correlations between successive layers to further retain predictive power of the compact network. Training with OICSR thoroughly transfers discriminative features into a fraction of out-in-channels. Correspondingly, OICSR measures channel importance based on statistics computed from two consecutive layers, not individual layer. Finally, a global greedy pruning algorithm is designed to remove redundant out-in-channels in an iterative way. Our method is comprehensively evaluated with various CNN architectures including CifarNet, AlexNet, ResNet, DenseNet and PreActSeNet on CIFAR-10, CIFAR-100 and ImageNet-1K datasets. Notably, on ImageNet-1K, we reduce 37.2% FLOPs on ResNet-50 while outperforming the original model by 0.22% top-1 accuracy.

arXiv commentAccepted to CVPR 2019, the pruned ResNet-50 model has be released at: https://github.com/dsfour/OICSR

References in corpus (7)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Distilling the Knowledge in a Neural Network
• R-FCN: Object Detection via Region-based Fully Convolutional Networks
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• N2N Learning: Network to Network Compression via Policy Gradient Reinforcement Learning