Layer-compensated Pruning for Resource-constrained Convolutional Neural Networks

Abstract

Resource-efficient convolution neural networks enable not only the intelligence on edge devices but also opportunities in system-level optimization such as scheduling. In this work, we aim to improve the performance of resource-constrained filter pruning by merging two sub-problems commonly considered, i.e., (i) how many filters to prune for each layer and (ii) which filters to prune given a per-layer pruning budget, into a global filter ranking problem. Our framework entails a novel algorithm, dubbed layer-compensated pruning, where meta-learning is involved to determine better solutions. We show empirically that the proposed algorithm is superior to prior art in both effectiveness and efficiency. Specifically, we reduce the accuracy gap between the pruned and original networks from 0.9% to 0.7% with 8x reduction in time needed for meta-learning, i.e., from 1 hour down to 7 minutes. To this end, we demonstrate the effectiveness of our algorithm using ResNet and MobileNetV2 networks under CIFAR-10, ImageNet, and Bird-200 datasets.

arXiv comment11 pages, 8 figures, work in progress

References in corpus (10)

• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
• Bayesian Compression for Deep Learning
• Exploring the Regularity of Sparse Structure in Convolutional Neural Networks
• Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers
• MONAS: Multi-Objective Neural Architecture Search using Reinforcement Learning
• Faster gaze prediction with dense networks and Fisher pruning
• Compressing Neural Networks using the Variational Information Bottleneck
• Structured Probabilistic Pruning for Convolutional Neural Network Acceleration
• Energy-Constrained Compression for Deep Neural Networks via Weighted Sparse Projection and Layer Input Masking

Cited by in corpus (15)

• AutoML: A Survey of the State-of-the-Art
• Filter Pruning by Switching to Neighboring CNNs with Good Attributes
• PoPS: Policy Pruning and Shrinking for Deep Reinforcement Learning
• Hybrid Pruning: Thinner Sparse Networks for Fast Inference on Edge Devices
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• LeanConvNets: Low-cost Yet Effective Convolutional Neural Networks
• Shapley Value as Principled Metric for Structured Network Pruning
• Dynamic Neural Network Channel Execution for Efficient Training
• Single-Path Mobile AutoML: Efficient ConvNet Design and NAS Hyperparameter Optimization
• Dynamic Sparse Graph for Efficient Deep Learning
• SASL: Saliency-Adaptive Sparsity Learning for Neural Network Acceleration
• Supervised Robustness-preserving Data-free Neural Network Pruning
• Filter Pruning using Hierarchical Group Sparse Regularization for Deep Convolutional Neural Networks
• New Directions in Distributed Deep Learning: Bringing the Network at Forefront of IoT Design
• Pruning Ternary Quantization