SASL: Saliency-Adaptive Sparsity Learning for Neural Network Acceleration

Abstract

Accelerating the inference speed of CNNs is critical to their deployment in real-world applications. Among all the pruning approaches, those implementing a sparsity learning framework have shown to be effective as they learn and prune the models in an end-to-end data-driven manner. However, these works impose the same sparsity regularization on all filters indiscriminately, which can hardly result in an optimal structure-sparse network. In this paper, we propose a Saliency-Adaptive Sparsity Learning (SASL) approach for further optimization. A novel and effective estimation of each filter, i.e., saliency, is designed, which is measured from two aspects: the importance for the prediction performance and the consumed computational resources. During sparsity learning, the regularization strength is adjusted according to the saliency, so our optimized format can better preserve the prediction performance while zeroing out more computation-heavy filters. The calculation for saliency introduces minimum overhead to the training process, which means our SASL is very efficient. During the pruning phase, in order to optimize the proposed data-dependent criterion, a hard sample mining strategy is utilized, which shows higher effectiveness and efficiency. Extensive experiments demonstrate the superior performance of our method. Notably, on ILSVRC-2012 dataset, our approach can reduce 49.7% FLOPs of ResNet-50 with very negligible 0.39% top-1 and 0.05% top-5 accuracy degradation.

arXiv commentAccepted to IEEE Transactions on Circuits and Systems for Video Technology

References in corpus (14)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• Understanding deep learning requires rethinking generalization
• Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
• Pruning Filters for Efficient ConvNets
• Variational Dropout Sparsifies Deep Neural Networks
• Global Sparse Momentum SGD for Pruning Very Deep Neural Networks
• MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning
• Accelerator-Aware Pruning for Convolutional Neural Networks
• Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks
• Layer-compensated Pruning for Resource-constrained Convolutional Neural Networks
• DeepHoyer: Learning Sparser Neural Network with Differentiable Scale-Invariant Sparsity Measures