A Main/Subsidiary Network Framework for Simplifying Binary Neural Network
arXiv:1812.04210
Abstract
To reduce memory footprint and run-time latency, techniques such as neural network pruning and binarization have been explored separately. However, it is unclear how to combine the best of the two worlds to get extremely small and efficient models. In this paper, we, for the first time, define the filter-level pruning problem for binary neural networks, which cannot be solved by simply migrating existing structural pruning methods for full-precision models. A novel learning-based approach is proposed to prune filters in our main/subsidiary network framework, where the main network is responsible for learning representative features to optimize the prediction performance, and the subsidiary component works as a filter selector on the main network. To avoid gradient mismatch when training the subsidiary component, we propose a layer-wise and bottom-up scheme. We also provide the theoretical and experimental comparison between our learning-based and greedy rule-based methods. Finally, we empirically demonstrate the effectiveness of our approach applied on several binary models, including binarized NIN, VGG-11, and ResNet-18, on various image classification datasets.
9 pages and 9 figures
References in corpus (12)
- Distilling the Knowledge in a Neural Network
- Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- BinaryConnect: Training Deep Neural Networks with binary weights during propagations
- Deep Learning with Limited Numerical Precision
- Trained Ternary Quantization
- Predicting Parameters in Deep Learning
- Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
- Channel Pruning for Accelerating Very Deep Neural Networks
- 8-Bit Approximations for Parallelism in Deep Learning
- Training Quantized Nets: A Deeper Understanding
- Deep Learning with Low Precision by Half-wave Gaussian Quantization