Provable Filter Pruning for Efficient Neural Networks

Abstract

We present a provable, sampling-based approach for generating compact Convolutional Neural Networks (CNNs) by identifying and removing redundant filters from an over-parameterized network. Our algorithm uses a small batch of input data points to assign a saliency score to each filter and constructs an importance sampling distribution where filters that highly affect the output are sampled with correspondingly high probability. In contrast to existing filter pruning approaches, our method is simultaneously data-informed, exhibits provable guarantees on the size and performance of the pruned network, and is widely applicable to varying network architectures and data sets. Our analytical bounds bridge the notions of compressibility and importance of network structures, which gives rise to a fully-automated procedure for identifying and preserving filters in layers that are essential to the network's performance. Our experimental evaluations on popular architectures and data sets show that our algorithm consistently generates sparser and more efficient models than those constructed by existing filter pruning approaches.

References in corpus (13)

• Wide Residual Networks
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Rethinking the Value of Network Pruning
• Pruning Filters for Efficient ConvNets
• Compressing Neural Networks with the Hashing Trick
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• The State of Sparsity in Deep Neural Networks
• Compression-aware Training of Deep Networks
• Dynamic Model Pruning with Feedback
• Binary embeddings with structured hashed projections
• SiPPing Neural Networks: Sensitivity-informed Provable Pruning of Neural Networks
• A Note on Randomized Element-wise Matrix Sparsification
• Revisiting hard thresholding for DNN pruning

Cited by in corpus (7)

• GhostNets on Heterogeneous Devices via Cheap Operations
• Good Subnetworks Provably Exist: Pruning via Greedy Forward Selection
• CHIP: CHannel Independence-based Pruning for Compact Neural Networks
• Exploring Gradient Flow Based Saliency for DNN Model Compression
• A Feature-map Discriminant Perspective for Pruning Deep Neural Networks
• Group and Exclusive Sparse Regularization-based Continual Learning of CNNs
• Low-Regret Active learning