Accelerate CNN via Recursive Bayesian Pruning

Abstract

Channel Pruning, widely used for accelerating Convolutional Neural Networks, is an NP-hard problem due to the inter-layer dependency of channel redundancy. Existing methods generally ignored the above dependency for computation simplicity. To solve the problem, under the Bayesian framework, we here propose a layer-wise Recursive Bayesian Pruning method (RBP). A new dropout-based measurement of redundancy, which facilitate the computation of posterior assuming inter-layer dependency, is introduced. Specifically, we model the noise across layers as a Markov chain and target its posterior to reflect the inter-layer dependency. Considering the closed form solution for posterior is intractable, we derive a sparsity-inducing Dirac-like prior which regularizes the distribution of the designed noise to automatically approximate the posterior. Compared with the existing methods, no additional overhead is required when the inter-layer dependency assumed. The redundant channels can be simply identified by tiny dropout noise and directly pruned layer by layer. Experiments on popular CNN architectures have shown that the proposed method outperforms several state-of-the-arts. Particularly, we achieve up to $5.0\times$ and $2.2\times$ FLOPs reduction with little accuracy loss on the large scale dataset ILSVRC2012 for VGG16 and ResNet50, respectively.

References in corpus (11)

• Improving neural networks by preventing co-adaptation of feature detectors
• Neural Architecture Search with Reinforcement Learning
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Weight Uncertainty in Neural Networks
• cuDNN: Efficient Primitives for Deep Learning
• Compressing Neural Networks with the Hashing Trick
• Bayesian Compression for Deep Learning
• Learning the Number of Neurons in Deep Networks
• Structured Bayesian Pruning via Log-Normal Multiplicative Noise
• Structured Transforms for Small-Footprint Deep Learning

Cited by in corpus (3)

• Group Sparsity: The Hinge Between Filter Pruning and Decomposition for Network Compression
• Cascaded channel pruning using hierarchical self-distillation
• Manipulating Identical Filter Redundancy for Efficient Pruning on Deep and Complicated CNN