Weight-dependent Gates for Network Pruning

Abstract

In this paper, a simple yet effective network pruning framework is proposed to simultaneously address the problems of pruning indicator, pruning ratio, and efficiency constraint. This paper argues that the pruning decision should depend on the convolutional weights, and thus proposes novel weight-dependent gates (W-Gates) to learn the information from filter weights and obtain binary gates to prune or keep the filters automatically. To prune the network under efficiency constraints, a switchable Efficiency Module is constructed to predict the hardware latency or FLOPs of candidate pruned networks. Combined with the proposed Efficiency Module, W-Gates can perform filter pruning in an efficiency-aware manner and achieve a compact network with a better accuracy-efficiency trade-off. We have demonstrated the effectiveness of the proposed method on ResNet34, ResNet50, and MobileNet V2, respectively achieving up to 1.33/1.28/1.1 higher Top-1 accuracy with lower hardware latency on ImageNet. Compared with state-of-the-art methods, W-Gates also achieves superior performance.

arXiv comment13 pages, 7 figures, to appear at IEEE TCSVT

References in corpus (15)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Categorical Reparameterization with Gumbel-Softmax
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Cost-Effective Active Learning for Deep Image Classification
• Compressing Neural Networks with the Hashing Trick
• Slimmable Neural Networks
• DeepCache: Principled Cache for Mobile Deep Vision
• Accelerator-Aware Pruning for Convolutional Neural Networks
• Video Semantic Segmentation with Distortion-Aware Feature Correction
• Approximated Oracle Filter Pruning for Destructive CNN Width Optimization
• Latency-Aware Differentiable Neural Architecture Search