Exploring Gradient Flow Based Saliency for DNN Model Compression

Abstract

Model pruning aims to reduce the deep neural network (DNN) model size or computational overhead. Traditional model pruning methods such as l-1 pruning that evaluates the channel significance for DNN pay too much attention to the local analysis of each channel and make use of the magnitude of the entire feature while ignoring its relevance to the batch normalization (BN) and ReLU layer after each convolutional operation. To overcome these problems, we propose a new model pruning method from a new perspective of gradient flow in this paper. Specifically, we first theoretically analyze the channel's influence based on Taylor expansion by integrating the effects of BN layer and ReLU activation function. Then, the incorporation of the first-order Talyor polynomial of the scaling parameter and the shifting parameter in the BN layer is suggested to effectively indicate the significance of a channel in a DNN. Comprehensive experiments on both image classification and image denoising tasks demonstrate the superiority of the proposed novel theory and scheme. Code is available at https://github.com/CityU-AIM-Group/GFBS.

ACM Multimedia 2021 (ACM MM'21)

References in corpus (28)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising
• MobileNetV2: Inverted Residuals and Linear Bottlenecks
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• Rethinking the Value of Network Pruning
• Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
• Pruning Filters for Efficient ConvNets
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Variational Dropout and the Local Reparameterization Trick
• Channel Pruning for Accelerating Very Deep Neural Networks
• ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware
• Learning Efficient Convolutional Networks through Network Slimming
• Network Pruning via Transformable Architecture Search
• MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning
• SCOP: Scientific Control for Reliable Neural Network Pruning
• DAIS: Automatic Channel Pruning via Differentiable Annealing Indicator Search
• More is Less: A More Complicated Network with Less Inference Complexity
• Towards Optimal Structured CNN Pruning via Generative Adversarial Learning
• Learning to Prune Filters in Convolutional Neural Networks
• Provable Filter Pruning for Efficient Neural Networks
• Group Sparsity: The Hinge Between Filter Pruning and Decomposition for Network Compression
• EagleEye: Fast Sub-net Evaluation for Efficient Neural Network Pruning
• DHP: Differentiable Meta Pruning via HyperNetworks
• DSA: More Efficient Budgeted Pruning via Differentiable Sparsity Allocation