Exploiting Channel Similarity for Accelerating Deep Convolutional Neural Networks
arXiv:1908.02620
Abstract
To address the limitations of existing magnitude-based pruning algorithms in cases where model weights or activations are of large and similar magnitude, we propose a novel perspective to discover parameter redundancy among channels and accelerate deep CNNs via channel pruning. Precisely, we argue that channels revealing similar feature information have functional overlap and that most channels within each such similarity group can be removed without compromising model's representational power. After deriving an effective metric for evaluating channel similarity through probabilistic modeling, we introduce a pruning algorithm via hierarchical clustering of channels. In particular, the proposed algorithm does not rely on sparsity training techniques or complex data-driven optimization and can be directly applied to pre-trained models. Extensive experiments on benchmark datasets strongly demonstrate the superior acceleration performance of our approach over prior arts. On ImageNet, our pruned ResNet-50 with 30% FLOPs reduced outperforms the baseline model.
arXiv comment14 pages, 6 figures
References in corpus (16)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Distilling the Knowledge in a Neural Network
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Neural Architecture Search with Reinforcement Learning
- Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
- Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
- Do Deep Nets Really Need to be Deep?
- Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
- DARTS: Differentiable Architecture Search
- Rethinking the Value of Network Pruning
- Large-Scale Evolution of Image Classifiers
- Neural Architecture Optimization
- Designing Neural Network Architectures using Reinforcement Learning
- Variational Dropout Sparsifies Deep Neural Networks
- Compressing Neural Networks using the Variational Information Bottleneck
- Diversity Networks: Neural Network Compression Using Determinantal Point Processes