Channel-wise pruning of neural networks with tapering resource constraint

Abstract

Neural network pruning is an important step in design process of efficient neural networks for edge devices with limited computational power. Pruning is a form of knowledge transfer from the weights of the original network to a smaller target subnetwork. We propose a new method for compute-constrained structured channel-wise pruning of convolutional neural networks. The method iteratively fine-tunes the network, while gradually tapering the computation resources available to the pruned network via a holonomic constraint in the method of Lagrangian multipliers framework. An explicit and adaptive automatic control over the rate of tapering is provided. The trainable parameters of our pruning method are separate from the weights of the neural network, which allows us to avoid the interference with the neural network solver (e.g. avoid the direct dependence of pruning speed on neural network learning rates). Our method combines the `rigoristic' approach by the direct application of constrained optimization, avoiding the pitfalls of ADMM-based methods, like their need to define the target amount of resources for each pruning run, and direct dependence of pruning speed and priority of pruning on the relative scale of weights between layers. For VGG-16 @ ILSVRC-2012, we achieve reduction of 15.47 -> 3.87 GMAC with only 1% top-1 accuracy reduction (68.4% -> 67.4%). For AlexNet @ ILSVRC-2012, we achieve 0.724 -> 0.411 GMAC with 1% top-1 accuracy reduction (56.8% -> 55.8%).

References in corpus (16)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
• Pruning Filters for Efficient ConvNets
• A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers
• Exploring the Regularity of Sparse Structure in Convolutional Neural Networks
• Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers
• Dynamic Network Surgery for Efficient DNNs
• Faster CNNs with Direct Sparse Convolutions and Guided Pruning
• Faster gaze prediction with dense networks and Fisher pruning
• Structured Probabilistic Pruning for Convolutional Neural Network Acceleration
• Model compression as constrained optimization, with application to neural nets. Part I: general framework
• Systematic Weight Pruning of DNNs using Alternating Direction Method of Multipliers
• The Incredible Shrinking Neural Network: New Perspectives on Learning Representations Through The Lens of Pruning
• Asymptotic Soft Filter Pruning for Deep Convolutional Neural Networks
• Towards thinner convolutional neural networks through Gradually Global Pruning