A novel channel pruning method for deep neural network compression

Abstract

In recent years, deep neural networks have achieved great success in the field of computer vision. However, it is still a big challenge to deploy these deep models on resource-constrained embedded devices such as mobile robots, smart phones and so on. Therefore, network compression for such platforms is a reasonable solution to reduce memory consumption and computation complexity. In this paper, a novel channel pruning method based on genetic algorithm is proposed to compress very deep Convolution Neural Networks (CNNs). Firstly, a pre-trained CNN model is pruned layer by layer according to the sensitivity of each layer. After that, the pruned model is fine-tuned based on knowledge distillation framework. These two improvements significantly decrease the model redundancy with less accuracy drop. Channel selection is a combinatorial optimization problem that has exponential solution space. In order to accelerate the selection process, the proposed method formulates it as a search problem, which can be solved efficiently by genetic algorithm. Meanwhile, a two-step approximation fitness function is designed to further improve the efficiency of genetic process. The proposed method has been verified on three benchmark datasets with two popular CNN models: VGGNet and ResNet. On the CIFAR-100 and ImageNet datasets, our approach outperforms several state-of-the-art methods. On the CIFAR-10 and SVHN datasets, the pruned VGGNet achieves better performance than the original model with 8 times parameters compression and 3 times FLOPs reduction.

References in corpus (8)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
• Compressing Neural Networks with the Hashing Trick
• Channel Pruning for Accelerating Very Deep Neural Networks
• Exploring the Regularity of Sparse Structure in Convolutional Neural Networks

Cited by in corpus (9)

• A Survey on Evolutionary Neural Architecture Search
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• GeneCAI: Genetic Evolution for Acquiring Compact AI
• Accelerating Deep Unsupervised Domain Adaptation with Transfer Channel Pruning
• Dynamic Sparse Graph for Efficient Deep Learning
• Manipulating Identical Filter Redundancy for Efficient Pruning on Deep and Complicated CNN
• On the Orthogonality of Knowledge Distillation with Other Techniques: From an Ensemble Perspective
• Batch Normalization Sampling
• ASCAI: Adaptive Sampling for acquiring Compact AI