SCOP: Scientific Control for Reliable Neural Network Pruning

Abstract

This paper proposes a reliable neural network pruning algorithm by setting up a scientific control. Existing pruning methods have developed various hypotheses to approximate the importance of filters to the network and then execute filter pruning accordingly. To increase the reliability of the results, we prefer to have a more rigorous research design by including a scientific control group as an essential part to minimize the effect of all factors except the association between the filter and expected network output. Acting as a control group, knockoff feature is generated to mimic the feature map produced by the network filter, but they are conditionally independent of the example label given the real feature map. We theoretically suggest that the knockoff condition can be approximately preserved given the information propagation of network layers. Besides the real feature map on an intermediate layer, the corresponding knockoff feature is brought in as another auxiliary input signal for the subsequent layers. Redundant filters can be discovered in the adversarial process of different features. Through experiments, we demonstrate the superiority of the proposed algorithm over state-of-the-art methods. For example, our method can reduce 57.8% parameters and 60.2% FLOPs of ResNet-101 with only 0.01% top-1 accuracy loss on ImageNet. The code is available at https://github.com/huawei-noah/Pruning/tree/master/SCOP_NeurIPS2020.

arXiv commentThis paper is accepted by NeurIPS 2020. Key words: Filter Pruning, Adversarial Pruning, Network Compression, CNN, Deep Learning

References in corpus (7)

• Adam: A Method for Stochastic Optimization
• AutoGAN-Distiller: Searching to Compress Generative Adversarial Networks
• Training Binary Neural Networks through Learning with Noisy Supervision
• Searching for Low-Bit Weights in Quantized Neural Networks
• Positive-Unlabeled Compression on the Cloud
• Bringing Giant Neural Networks Down to Earth with Unlabeled Data
• AdaDeep: A Usage-Driven, Automated Deep Model Compression Framework for Enabling Ubiquitous Intelligent Mobiles

Cited by in corpus (13)

• Only Train Once: A One-Shot Neural Network Training And Pruning Framework
• LOss-Based SensiTivity rEgulaRization: towards deep sparse neural networks
• Distilling Object Detectors via Decoupled Features
• Pruning-aware Sparse Regularization for Network Pruning
• HALOC: Hardware-Aware Automatic Low-Rank Compression for Compact Neural Networks
• Feature Flow Regularization: Improving Structured Sparsity in Deep Neural Networks
• Packet Header Recognition Utilizing an All-Optical Reservoir Based on Reinforcement-Learning-Optimized Double-Ring Resonator
• Learning Versatile Convolution Filters for Efficient Visual Recognition
• Exploring Gradient Flow Based Saliency for DNN Model Compression
• Manifold Regularized Dynamic Network Pruning
• Supervised Robustness-preserving Data-free Neural Network Pruning
• TDC: Towards Extremely Efficient CNNs on GPUs via Hardware-Aware Tucker Decomposition
• BCNet: Searching for Network Width with Bilaterally Coupled Network