Towards Optimal Structured CNN Pruning via Generative Adversarial Learning

Abstract

Structured pruning of filters or neurons has received increased focus for compressing convolutional neural networks. Most existing methods rely on multi-stage optimizations in a layer-wise manner for iteratively pruning and retraining which may not be optimal and may be computation intensive. Besides, these methods are designed for pruning a specific structure, such as filter or block structures without jointly pruning heterogeneous structures. In this paper, we propose an effective structured pruning approach that jointly prunes filters as well as other structures in an end-to-end manner. To accomplish this, we first introduce a soft mask to scale the output of these structures by defining a new objective function with sparsity regularization to align the output of baseline and network with this mask. We then effectively solve the optimization problem by generative adversarial learning (GAL), which learns a sparse soft mask in a label-free and an end-to-end manner. By forcing more scaling factors in the soft mask to zero, the fast iterative shrinkage-thresholding algorithm (FISTA) can be leveraged to fast and reliably remove the corresponding structures. Extensive experiments demonstrate the effectiveness of GAL on different datasets, including MNIST, CIFAR-10 and ImageNet ILSVRC 2012. For example, on ImageNet ILSVRC 2012, the pruned ResNet-50 achieves 10.88\% Top-5 error and results in a factor of 3.7x speedup. This significantly outperforms state-of-the-art methods.

Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

References in corpus (12)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Conditional Generative Adversarial Nets
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
• Designing Neural Network Architectures using Reinforcement Learning
• A Field Guide to Forward-Backward Splitting with a FASTA Implementation
• Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration
• Towards Compact ConvNets via Structure-Sparsity Regularized Filter Pruning

Cited by in corpus (12)

• A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
• Model Rubik's Cube: Twisting Resolution, Depth and Width for TinyNets
• DAIS: Automatic Channel Pruning via Differentiable Annealing Indicator Search
• CARS: Continuous Evolution for Efficient Neural Architecture Search
• Locally Free Weight Sharing for Network Width Search
• DBP: Discrimination Based Block-Level Pruning for Deep Model Acceleration
• Accelerate CNNs from Three Dimensions: A Comprehensive Pruning Framework
• DMCP: Differentiable Markov Channel Pruning for Neural Networks
• PAMS: Quantized Super-Resolution via Parameterized Max Scale
• Learning Versatile Convolution Filters for Efficient Visual Recognition
• Exploring Gradient Flow Based Saliency for DNN Model Compression
• Cogradient Descent for Dependable Learning