Gradual Channel Pruning while Training using Feature Relevance Scores for Convolutional Neural Networks

Abstract

The enormous inference cost of deep neural networks can be scaled down by network compression. Pruning is one of the predominant approaches used for deep network compression. However, existing pruning techniques have one or more of the following limitations: 1) Additional energy cost on top of the compute heavy training stage due to pruning and fine-tuning stages, 2) Layer-wise pruning based on the statistics of a particular, ignoring the effect of error propagation in the network, 3) Lack of an efficient estimate for determining the important channels globally, 4) Unstructured pruning requires specialized hardware for effective use. To address all the above issues, we present a simple-yet-effective gradual channel pruning while training methodology using a novel data-driven metric referred to as feature relevance score. The proposed technique gets rid of the additional retraining cycles by pruning the least important channels in a structured fashion at fixed intervals during the actual training phase. Feature relevance scores help in efficiently evaluating the contribution of each channel towards the discriminative power of the network. We demonstrate the effectiveness of the proposed methodology on architectures such as VGG and ResNet using datasets such as CIFAR-10, CIFAR-100 and ImageNet, and successfully achieve significant model compression while trading off less than $1\%$ accuracy. Notably on CIFAR-10 dataset trained on ResNet-110, our approach achieves $2.4\times$ compression and a $56\%$ reduction in FLOPs with an accuracy drop of $0.01\%$ compared to the unpruned network.

arXiv comment15 pages, 2 figures, 4 tables

References in corpus (14)

• Explaining NonLinear Classification Decisions with Deep Taylor Decomposition
• Rethinking the Value of Network Pruning
• Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
• Predicting Parameters in Deep Learning
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• Learning Structured Sparsity in Deep Neural Networks
• Variational Dropout Sparsifies Deep Neural Networks
• Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers
• EIE: Efficient Inference Engine on Compressed Deep Neural Network
• Gate Decorator: Global Filter Pruning Method for Accelerating Deep Convolutional Neural Networks
• A Low Effort Approach to Structured CNN Design Using PCA
• Interpreting the Predictions of Complex ML Models by Layer-wise Relevance Propagation
• RePr: Improved Training of Convolutional Filters
• Really should we pruning after model be totally trained? Pruning based on a small amount of training

Cited by in corpus (1)

• When to Prune? A Policy towards Early Structural Pruning