Effective Model Compression via Stage-wise Pruning

Abstract

Automated Machine Learning(Auto-ML) pruning methods aim at searching a pruning strategy automatically to reduce the computational complexity of deep Convolutional Neural Networks(deep CNNs). However, some previous work found that the results of many Auto-ML pruning methods cannot even surpass the results of the uniformly pruning method. In this paper, the ineffectiveness of Auto-ML pruning which is caused by unfull and unfair training of the supernet is shown. A deep supernet suffers from unfull training because it contains too many candidates. To overcome the unfull training, a stage-wise pruning(SWP) method is proposed, which splits a deep supernet into several stage-wise supernets to reduce the candidate number and utilize inplace distillation to supervise the stage training. Besides, A wide supernet is hit by unfair training since the sampling probability of each channel is unequal. Therefore, the fullnet and the tinynet are sampled in each training iteration to ensure each channel can be overtrained. Remarkably, the proxy performance of the subnets trained with SWP is closer to the actual performance than that of most of the previous Auto-ML pruning work. Experiments show that SWP achieves the state-of-the-art on both CIFAR-10 and ImageNet under the mobile setting.

References in corpus (16)

• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• Neural Architecture Search with Reinforcement Learning
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• Efficient Neural Architecture Search via Parameter Sharing
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• Slimmable Neural Networks
• Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers
• AutoSlim: Towards One-Shot Architecture Search for Channel Numbers
• MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning
• Building Efficient ConvNets using Redundant Feature Pruning
• Approximated Oracle Filter Pruning for Destructive CNN Width Optimization
• Knowledge Projection for Deep Neural Networks