Discrimination-aware Network Pruning for Deep Model Compression

Abstract

We study network pruning which aims to remove redundant channels/kernels and hence speed up the inference of deep networks. Existing pruning methods either train from scratch with sparsity constraints or minimize the reconstruction error between the feature maps of the pre-trained models and the compressed ones. Both strategies suffer from some limitations: the former kind is computationally expensive and difficult to converge, while the latter kind optimizes the reconstruction error but ignores the discriminative power of channels. In this paper, we propose a simple-yet-effective method called discrimination-aware channel pruning (DCP) to choose the channels that actually contribute to the discriminative power. Note that a channel often consists of a set of kernels. Besides the redundancy in channels, some kernels in a channel may also be redundant and fail to contribute to the discriminative power of the network, resulting in kernel level redundancy. To solve this, we propose a discrimination-aware kernel pruning (DKP) method to further compress deep networks by removing redundant kernels. To prevent DCP/DKP from selecting redundant channels/kernels, we propose a new adaptive stopping condition, which helps to automatically determine the number of selected channels/kernels and often results in more compact models with better performance. Extensive experiments on both image classification and face recognition demonstrate the effectiveness of our methods. For example, on ILSVRC-12, the resultant ResNet-50 model with 30% reduction of channels even outperforms the baseline model by 0.36% in terms of Top-1 accuracy. The pruned MobileNetV1 and MobileNetV2 achieve 1.93x and 1.42x inference acceleration on a mobile device, respectively, with negligible performance degradation. The source code and the pre-trained models are available at https://github.com/SCUT-AILab/DCP.

arXiv comment14 pages. Extended version of the NeurIPS paper arXiv:1810.11809

References in corpus (35)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• FaceNet: A Unified Embedding for Face Recognition and Clustering
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Two-Stream Convolutional Networks for Action Recognition in Videos
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Training Very Deep Networks
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• Deeply-Supervised Nets
• Compressing Deep Convolutional Networks using Vector Quantization
• Rethinking the Value of Network Pruning
• Binarized Neural Networks
• DeepID3: Face Recognition with Very Deep Neural Networks
• Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
• PACT: Parameterized Clipping Activation for Quantized Neural Networks
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
• Learning Structured Sparsity in Deep Neural Networks
• Convolutional neural networks with low-rank regularization
• Discrimination-aware Channel Pruning for Deep Neural Networks
• Gate Decorator: Global Filter Pruning Method for Accelerating Deep Convolutional Neural Networks
• MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning
• Gradient Hard Thresholding Pursuit for Sparsity-Constrained Optimization
• NAT: Neural Architecture Transformer for Accurate and Compact Architectures
• Structured Probabilistic Pruning for Convolutional Neural Network Acceleration
• Forward-Backward Greedy Algorithms for General Convex Smooth Functions over A Cardinality Constraint
• Hierarchical binary CNNs for landmark localization with limited resources
• Adversarial Learning with Local Coordinate Coding
• Structured Binary Neural Networks for Accurate Image Classification and Semantic Segmentation
• Dual Reconstruction Nets for Image Super-Resolution with Gradient Sensitive Loss
• The Shallow End: Empowering Shallower Deep-Convolutional Networks through Auxiliary Outputs

Cited by in corpus (16)

• NetLLM: Adapting Large Language Models for Networking
• EPMF: Efficient Perception-aware Multi-sensor Fusion for 3D Semantic Segmentation
• Closed-loop Matters: Dual Regression Networks for Single Image Super-Resolution
• CHIP: CHannel Independence-based Pruning for Compact Neural Networks
• Towards Efficient Post-training Quantization of Pre-trained Language Models
• Optimizing Convolutional Neural Network Architecture
• Maximizing Discrimination Capability of Knowledge Distillation with Energy Function
• Deep Network Pruning: A Comparative Study on CNNs in Face Recognition
• Taxonomy of Saliency Metrics for Channel Pruning
• Finding Deviated Behaviors of the Compressed DNN Models for Image Classifications
• Diverse Sample Generation: Pushing the Limit of Generative Data-free Quantization
• Convolutional Neural Network Simplification with Progressive Retraining
• Improved Techniques for the Conditional Generative Augmentation of Clinical Audio Data
• Rapid Elastic Architecture Search under Specialized Classes and Resource Constraints
• Pruning with Compensation: Efficient Channel Pruning for Deep Convolutional Neural Networks
• Content-Aware Convolutional Neural Networks