Approximated Oracle Filter Pruning for Destructive CNN Width Optimization

Abstract

It is not easy to design and run Convolutional Neural Networks (CNNs) due to: 1) finding the optimal number of filters (i.e., the width) at each layer is tricky, given an architecture; and 2) the computational intensity of CNNs impedes the deployment on computationally limited devices. Oracle Pruning is designed to remove the unimportant filters from a well-trained CNN, which estimates the filters' importance by ablating them in turn and evaluating the model, thus delivers high accuracy but suffers from intolerable time complexity, and requires a given resulting width but cannot automatically find it. To address these problems, we propose Approximated Oracle Filter Pruning (AOFP), which keeps searching for the least important filters in a binary search manner, makes pruning attempts by masking out filters randomly, accumulates the resulting errors, and finetunes the model via a multi-path framework. As AOFP enables simultaneous pruning on multiple layers, we can prune an existing very deep CNN with acceptable time cost, negligible accuracy drop, and no heuristic knowledge, or re-design a model which exerts higher accuracy and faster inference.

ICML 2019

Cited by in corpus (17)

• ACNet: Strengthening the Kernel Skeletons for Powerful CNN via Asymmetric Convolution Blocks
• RepVGG: Making VGG-style ConvNets Great Again
• Weight-dependent Gates for Network Pruning
• DMCP: Differentiable Markov Channel Pruning for Neural Networks
• Convolutional Neural Network Pruning with Structural Redundancy Reduction
• PENNI: Pruned Kernel Sharing for Efficient CNN Inference
• Towards Compact CNNs via Collaborative Compression
• Composition of Saliency Metrics for Channel Pruning with a Myopic Oracle
• Filter Sketch for Network Pruning
• Model compression as constrained optimization, with application to neural nets. Part V: combining compressions
• Neural Pruning via Growing Regularization
• Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition
• Network Pruning using Adaptive Exemplar Filters
• Joint Channel and Weight Pruning for Model Acceleration on Moblie Devices
• Graph Pruning for Model Compression
• Effective Model Compression via Stage-wise Pruning
• Dep-$L_0$: Improving $L_0$-based Network Sparsification via Dependency Modeling