Centripetal SGD for Pruning Very Deep Convolutional Networks with Complicated Structure
arXiv:1904.03837
Abstract
The redundancy is widely recognized in Convolutional Neural Networks (CNNs), which enables to remove unimportant filters from convolutional layers so as to slim the network with acceptable performance drop. Inspired by the linear and combinational properties of convolution, we seek to make some filters increasingly close and eventually identical for network slimming. To this end, we propose Centripetal SGD (C-SGD), a novel optimization method, which can train several filters to collapse into a single point in the parameter hyperspace. When the training is completed, the removal of the identical filters can trim the network with NO performance loss, thus no finetuning is needed. By doing so, we have partly solved an open problem of constrained filter pruning on CNNs with complicated structure, where some layers must be pruned following others. Our experimental results on CIFAR-10 and ImageNet have justified the effectiveness of C-SGD-based filter pruning. Moreover, we have provided empirical evidences for the assumption that the redundancy in deep neural networks helps the convergence of training by showing that a redundant CNN trained using C-SGD outperforms a normally trained counterpart with the equivalent width.
CVPR 2019
References in corpus (24)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Distilling the Knowledge in a Neural Network
- TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
- Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
- FitNets: Hints for Thin Deep Nets
- Do Deep Nets Really Need to be Deep?
- Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
- Pruning Convolutional Neural Networks for Resource Efficient Inference
- Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
- Predicting Parameters in Deep Learning
- Pruning Filters for Efficient ConvNets
- Speeding up Convolutional Neural Networks with Low Rank Expansions
- A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers
- Fast Training of Convolutional Networks through FFTs
- Fast Convolutional Nets With fbfft: A GPU Performance Evaluation
- Learning the Number of Neurons in Deep Networks
- Memory Bounded Deep Convolutional Networks
- PerforatedCNNs: Acceleration through Elimination of Redundant Convolutions
- Structured Transforms for Small-Footprint Deep Learning
- Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks
- Structural Compression of Convolutional Neural Networks
- Towards Compact ConvNets via Structure-Sparsity Regularized Filter Pruning
- Structured Pruning for Efficient ConvNets via Incremental Regularization