Pruning Filter in Filter

Abstract

Pruning has become a very powerful and effective technique to compress and accelerate modern neural networks. Existing pruning methods can be grouped into two categories: filter pruning (FP) and weight pruning (WP). FP wins at hardware compatibility but loses at the compression ratio compared with WP. To converge the strength of both methods, we propose to prune the filter in the filter. Specifically, we treat a filter $F\in {\mathbb{R}}^{C\times K\times K}$ as $K\times K$ stripes, i.e., $1\times 1$ filters $\in {\mathbb{R}}^C$, then by pruning the stripes instead of the whole filter, we can achieve finer granularity than traditional FP while being hardware friendly. We term our method as SWP (\emph{Stripe-Wise Pruning}). SWP is implemented by introducing a novel learnable matrix called Filter Skeleton, whose values reflect the shape of each filter. As some recent work has shown that the pruned architecture is more crucial than the inherited important weights, we argue that the architecture of a single filter, i.e., the shape, also matters. Through extensive experiments, we demonstrate that SWP is more effective compared to the previous FP-based methods and achieves the state-of-art pruning ratio on CIFAR-10 and ImageNet datasets without obvious accuracy drop. Code is available at https://github.com/fxmeng/Pruning-Filter-in-Filter

arXiv commentAccepted by NeurIPS2020

References in corpus (16)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• Rethinking the Value of Network Pruning
• Pruning Filters for Efficient ConvNets
• Text Understanding from Scratch
• Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning
• Gate Decorator: Global Filter Pruning Method for Accelerating Deep Convolutional Neural Networks
• Soft Threshold Weight Reparameterization for Learnable Sparsity
• Accelerator-Aware Pruning for Convolutional Neural Networks
• Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks
• Shift: A Zero FLOP, Zero Parameter Alternative to Spatial Convolutions
• Group Sparsity: The Hinge Between Filter Pruning and Decomposition for Network Compression
• DMCP: Differentiable Markov Channel Pruning for Neural Networks
• HRank: Filter Pruning using High-Rank Feature Map
• Full-Stack Filters to Build Minimum Viable CNNs

Cited by in corpus (2)

• Only Train Once: A One-Shot Neural Network Training And Pruning Framework
• Learning Pruned Structure and Weights Simultaneously from Scratch: an Attention based Approach