Learning Efficient Convolutional Networks through Network Slimming

Abstract

The deployment of deep convolutional neural networks (CNNs) in many real world applications is largely hindered by their high computational cost. In this paper, we propose a novel learning scheme for CNNs to simultaneously 1) reduce the model size; 2) decrease the run-time memory footprint; and 3) lower the number of computing operations, without compromising accuracy. This is achieved by enforcing channel-level sparsity in the network in a simple but effective way. Different from many existing approaches, the proposed method directly applies to modern CNN architectures, introduces minimum overhead to the training process, and requires no special software/hardware accelerators for the resulting models. We call our approach network slimming, which takes wide and large networks as input models, but during training insignificant channels are automatically identified and pruned afterwards, yielding thin and compact models with comparable accuracy. We empirically demonstrate the effectiveness of our approach with several state-of-the-art CNN models, including VGGNet, ResNet and DenseNet, on various image classification datasets. For VGGNet, a multi-pass version of network slimming gives a 20x reduction in model size and a 5x reduction in computing operations.

Accepted by ICCV 2017

References in corpus (9)

• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Pruning Filters for Efficient ConvNets
• Group Sparse Regularization for Deep Neural Networks
• Learning Structured Sparsity in Deep Neural Networks
• Deep Networks with Stochastic Depth
• The Power of Sparsity in Convolutional Neural Networks
• Training Sparse Neural Networks
• Neural Network Architecture Optimization through Submodularity and Supermodularity

Cited by in corpus (34)

• ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware
• MCUNet: Tiny Deep Learning on IoT Devices
• Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers
• Once-for-All: Train One Network and Specialize it for Efficient Deployment
• Review of data analysis in vision inspection of power lines with an in-depth discussion of deep learning technology
• What Do Compressed Deep Neural Networks Forget?
• MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning
• M2KD: Multi-model and Multi-level Knowledge Distillation for Incremental Learning
• Dynamic Neural Networks: A Survey
• Accelerate CNNs from Three Dimensions: A Comprehensive Pruning Framework
• DBP: Discrimination Based Block-Level Pruning for Deep Model Acceleration
• Hardware-Centric AutoML for Mixed-Precision Quantization
• Anytime Inference with Distilled Hierarchical Neural Ensembles
• GNN is a Counter? Revisiting GNN for Question Answering
• BLK-REW: A Unified Block-based DNN Pruning Framework using Reweighted Regularization Method
• Full deep neural network training on a pruned weight budget
• Exploring Gradient Flow Based Saliency for DNN Model Compression
• NaturalInversion: Data-Free Image Synthesis Improving Real-World Consistency
• Fractional Skipping: Towards Finer-Grained Dynamic CNN Inference
• Convolutional Neural Network Simplification with Progressive Retraining
• VACL: Variance-Aware Cross-Layer Regularization for Pruning Deep Residual Networks
• Block-wise Dynamic Sparseness
• Progressive Learning of Low-Precision Networks
• Learning Compact Representations of Neural Networks using DiscriminAtive Masking (DAM)
• Convolutional neural networks compression with low rank and sparse tensor decompositions
• AutoPruning for Deep Neural Network with Dynamic Channel Masking
• BN-NAS: Neural Architecture Search with Batch Normalization
• An Efficient Quantitative Approach for Optimizing Convolutional Neural Networks
• Now that I can see, I can improve: Enabling data-driven finetuning of CNNs on the edge
• Single Image Depth Prediction with Wavelet Decomposition
• Reliable Identification of Redundant Kernels for Convolutional Neural Network Compression
• WeClick: Weakly-Supervised Video Semantic Segmentation with Click Annotations
• Multi-Task Network Pruning and Embedded Optimization for Real-time Deployment in ADAS
• $S^2$-LBI: Stochastic Split Linearized Bregman Iterations for Parsimonious Deep Learning