Data-Driven Sparse Structure Selection for Deep Neural Networks
arXiv:1707.01213
Abstract
Deep convolutional neural networks have liberated its extraordinary power on various tasks. However, it is still very challenging to deploy state-of-the-art models into real-world applications due to their high computational complexity. How can we design a compact and effective network without massive experiments and expert knowledge? In this paper, we propose a simple and effective framework to learn and prune deep models in an end-to-end manner. In our framework, a new type of parameter -- scaling factor is first introduced to scale the outputs of specific structures, such as neurons, groups or residual blocks. Then we add sparsity regularizations on these factors, and solve this optimization problem by a modified stochastic Accelerated Proximal Gradient (APG) method. By forcing some of the factors to zero, we can safely remove the corresponding structures, thus prune the unimportant parts of a CNN. Comparing with other structure selection methods that may need thousands of trials or iterative fine-tuning, our method is trained fully end-to-end in one training pass without bells and whistles. We evaluate our method, Sparse Structure Selection with several state-of-the-art CNNs, and demonstrate very promising results with adaptive depth and width selection.
arXiv commentECCV Camera ready version
References in corpus (14)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Distilling the Knowledge in a Neural Network
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
- Neural Architecture Search with Reinforcement Learning
- Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
- MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
- Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
- Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
- Residual Networks Behave Like Ensembles of Relatively Shallow Networks
- Highway Networks
- Learning Structured Sparsity in Deep Neural Networks
- Channel Pruning for Accelerating Very Deep Neural Networks
- Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers
Cited by in corpus (6)
- Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers
- Compacting Deep Neural Networks for Internet of Things: Methods and Applications
- BlockDrop: Dynamic Inference Paths in Residual Networks
- Pruning at a Glance: Global Neural Pruning for Model Compression
- Learning Strict Identity Mappings in Deep Residual Networks
- Group Pruning using a Bounded-Lp norm for Group Gating and Regularization