Discrimination-aware Channel Pruning for Deep Neural Networks

Abstract

Channel pruning is one of the predominant approaches for deep model compression. Existing pruning methods either train from scratch with sparsity constraints on channels, or minimize the reconstruction error between the pre-trained feature maps and the compressed ones. Both strategies suffer from some limitations: the former kind is computationally expensive and difficult to converge, whilst the latter kind optimizes the reconstruction error but ignores the discriminative power of channels. To overcome these drawbacks, we investigate a simple-yet-effective method, called discrimination-aware channel pruning, to choose those channels that really contribute to discriminative power. To this end, we introduce additional losses into the network to increase the discriminative power of intermediate layers and then select the most discriminative channels for each layer by considering the additional loss and the reconstruction error. Last, we propose a greedy algorithm to conduct channel selection and parameter optimization in an iterative way. Extensive experiments demonstrate the effectiveness of our method. For example, on ILSVRC-12, our pruned ResNet-50 with 30% reduction of channels even outperforms the original model by 0.39% in top-1 accuracy.

arXiv commentNeurIPS 2018

References in corpus (21)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Two-Stream Convolutional Networks for Action Recognition in Videos
• MobileNetV2: Inverted Residuals and Linear Bottlenecks
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Learning Face Representation from Scratch
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Training Very Deep Networks
• Compressing Deep Convolutional Networks using Vector Quantization
• DeepID3: Face Recognition with Very Deep Neural Networks
• Ternary Weight Networks
• Trained Ternary Quantization
• Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
• Learning Structured Sparsity in Deep Neural Networks
• SphereFace: Deep Hypersphere Embedding for Face Recognition
• Channel Pruning for Accelerating Very Deep Neural Networks
• Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers
• Gradient Hard Thresholding Pursuit for Sparsity-Constrained Optimization
• Forward-Backward Greedy Algorithms for General Convex Smooth Functions over A Cardinality Constraint
• The Shallow End: Empowering Shallower Deep-Convolutional Networks through Auxiliary Outputs

Cited by in corpus (26)

• Dynamic Channel Pruning: Feature Boosting and Suppression
• Bringing AI To Edge: From Deep Learning's Perspective
• Discrimination-aware Network Pruning for Deep Model Compression
• DAIS: Automatic Channel Pruning via Differentiable Annealing Indicator Search
• Continual Learning with Node-Importance based Adaptive Group Sparse Regularization
• Task dependent Deep LDA pruning of neural networks
• CHIP: CHannel Independence-based Pruning for Compact Neural Networks
• HALOC: Hardware-Aware Automatic Low-Rank Compression for Compact Neural Networks
• SS-Auto: A Single-Shot, Automatic Structured Weight Pruning Framework of DNNs with Ultra-High Efficiency
• Bringing Giant Neural Networks Down to Earth with Unlabeled Data
• AdderNet: Do We Really Need Multiplications in Deep Learning?
• Fine-Grained Neural Architecture Search
• Rethinking Class-Discrimination Based CNN Channel Pruning
• Compact Global Descriptor for Neural Networks
• Convolutional Neural Network Simplification with Progressive Retraining
• Auxiliary Learning for Deep Multi-task Learning
• Hessian-Aware Pruning and Optimal Neural Implant
• Manifold Regularized Dynamic Network Pruning
• Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition
• FALCON: Lightweight and Accurate Convolution
• ThumbNet: One Thumbnail Image Contains All You Need for Recognition
• Intelligent Home 3D: Automatic 3D-House Design from Linguistic Descriptions Only
• BWCP: Probabilistic Learning-to-Prune Channels for ConvNets via Batch Whitening
• BCNet: Searching for Network Width with Bilaterally Coupled Network
• Dense Pruning of Pointwise Convolutions in the Frequency Domain
• Gator: Customizable Channel Pruning of Neural Networks with Gating