ChipNet: Budget-Aware Pruning with Heaviside Continuous Approximations

Abstract

Structured pruning methods are among the effective strategies for extracting small resource-efficient convolutional neural networks from their dense counterparts with minimal loss in accuracy. However, most existing methods still suffer from one or more limitations, that include 1) the need for training the dense model from scratch with pruning-related parameters embedded in the architecture, 2) requiring model-specific hyperparameter settings, 3) inability to include budget-related constraint in the training process, and 4) instability under scenarios of extreme pruning. In this paper, we present ChipNet, a deterministic pruning strategy that employs continuous Heaviside function and a novel crispness loss to identify a highly sparse network out of an existing dense network. Our choice of continuous Heaviside function is inspired by the field of design optimization, where the material distribution task is posed as a continuous optimization problem, but only discrete values (0 or 1) are practically feasible and expected as final outcomes. Our approach's flexible design facilitates its use with different choices of budget constraints while maintaining stability for very low target budgets. Experimental results show that ChipNet outperforms state-of-the-art structured pruning methods by remarkable margins of up to 16.1% in terms of accuracy. Further, we show that the masks obtained with ChipNet are transferable across datasets. For certain cases, it was observed that masks transferred from a model trained on feature-rich teacher dataset provide better performance on the student dataset than those obtained by directly pruning on the student data itself.

arXiv commentAccepted at ICLR 2021 Conference

References in corpus (10)

• Distilling the Knowledge in a Neural Network
• Decoupled Weight Decay Regularization
• Wide Residual Networks
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• Compressing Deep Convolutional Networks using Vector Quantization
• Pruning Filters for Efficient ConvNets
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• Learning the Number of Neurons in Deep Networks

Cited by in corpus (4)

• CHIP: CHannel Independence-based Pruning for Compact Neural Networks
• KinePose: A temporally optimized inverse kinematics technique for 6DOF human pose estimation with biomechanical constraints
• Layer Folding: Neural Network Depth Reduction using Activation Linearization
• Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition