ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression

Abstract

We propose an efficient and unified framework, namely ThiNet, to simultaneously accelerate and compress CNN models in both training and inference stages. We focus on the filter level pruning, i.e., the whole filter would be discarded if it is less important. Our method does not change the original network structure, thus it can be perfectly supported by any off-the-shelf deep learning libraries. We formally establish filter pruning as an optimization problem, and reveal that we need to prune filters based on statistics information computed from its next layer, not the current layer, which differentiates ThiNet from existing methods. Experimental results demonstrate the effectiveness of this strategy, which has advanced the state-of-the-art. We also show the performance of ThiNet on ILSVRC-12 benchmark. ThiNet achieves 3.31$\times$ FLOPs reduction and 16.63$\times$ compression on VGG-16, with only 0.52$\%$ top-5 accuracy drop. Similar experiments with ResNet-50 reveal that even for a compact network, ThiNet can also reduce more than half of the parameters and FLOPs, at the cost of roughly 1$\%$ top-5 accuracy drop. Moreover, the original VGG-16 model can be further pruned into a very small model with only 5.05MB model size, preserving AlexNet level accuracy but showing much stronger generalization ability.

To appear in ICCV 2017

References in corpus (13)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Improving neural networks by preventing co-adaptation of feature detectors
• Deep Residual Learning for Image Recognition
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Going Deeper with Convolutions
• Compressing Deep Convolutional Networks using Vector Quantization
• Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
• Predicting Parameters in Deep Learning
• Learning Deconvolution Network for Semantic Segmentation
• Learning Structured Sparsity in Deep Neural Networks
• Learning Deep Features for Discriminative Localization
• Quantized Convolutional Neural Networks for Mobile Devices
• Fast ConvNets Using Group-wise Brain Damage

Cited by in corpus (3)

• Full deep neural network training on a pruned weight budget
• On Iterative Neural Network Pruning, Reinitialization, and the Similarity of Masks
• Convolutional neural networks compression with low rank and sparse tensor decompositions