MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning

Abstract

In this paper, we propose a novel meta learning approach for automatic channel pruning of very deep neural networks. We first train a PruningNet, a kind of meta network, which is able to generate weight parameters for any pruned structure given the target network. We use a simple stochastic structure sampling method for training the PruningNet. Then, we apply an evolutionary procedure to search for good-performing pruned networks. The search is highly efficient because the weights are directly generated by the trained PruningNet and we do not need any finetuning at search time. With a single PruningNet trained for the target network, we can search for various Pruned Networks under different constraints with little human participation. Compared to the state-of-the-art pruning methods, we have demonstrated superior performances on MobileNet V1/V2 and ResNet. Codes are available on https://github.com/liuzechun/MetaPruning.

ICCV 2019 Camera ready version. Codes are available on https://github.com/liuzechun/MetaPruning

References in corpus (13)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Neural Architecture Search with Reinforcement Learning
• DARTS: Differentiable Architecture Search
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• Rethinking the Value of Network Pruning
• Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
• Large-Scale Evolution of Image Classifiers
• Learning the Number of Neurons in Deep Networks
• Loss-aware Weight Quantization of Deep Networks
• MetaAnchor: Learning to Detect Objects with Customized Anchors
• Meta-SR: A Magnification-Arbitrary Network for Super-Resolution
• Compact Deep Convolutional Neural Networks With Coarse Pruning

Cited by in corpus (37)

• Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better
• MCUNet: Tiny Deep Learning on IoT Devices
• Discrimination-aware Network Pruning for Deep Model Compression
• GhostNet: More Features from Cheap Operations
• DyNet: Dynamic Convolution for Accelerating Convolutional Neural Networks
• Good Subnetworks Provably Exist: Pruning via Greedy Forward Selection
• MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning
• Learning the Effect of Registration Hyperparameters with HyperMorph
• Principled Weight Initialization for Hypernetworks
• ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions
• Taxonomy and Evaluation of Structured Compression of Convolutional Neural Networks
• Generative Low-bitwidth Data Free Quantization
• EagleEye: Fast Sub-net Evaluation for Efficient Neural Network Pruning
• DHP: Differentiable Meta Pruning via HyperNetworks
• Highly Efficient Salient Object Detection with 100K Parameters
• DSA: More Efficient Budgeted Pruning via Differentiable Sparsity Allocation
• PENNI: Pruned Kernel Sharing for Efficient CNN Inference
• Learning Versatile Convolution Filters for Efficient Visual Recognition
• Exploring Gradient Flow Based Saliency for DNN Model Compression
• Distillation Guided Residual Learning for Binary Convolutional Neural Networks
• Training Compact CNNs for Image Classification using Dynamic-coded Filter Fusion
• Automatic Neural Network Compression by Sparsity-Quantization Joint Learning: A Constrained Optimization-based Approach
• Automated Model Compression by Jointly Applied Pruning and Quantization
• Filter Sketch for Network Pruning
• Network Pruning using Adaptive Exemplar Filters
• Greedy Optimization Provably Wins the Lottery: Logarithmic Number of Winning Tickets is Enough
• DS-Net++: Dynamic Weight Slicing for Efficient Inference in CNNs and Transformers
• SASL: Saliency-Adaptive Sparsity Learning for Neural Network Acceleration
• AdaPruner: Adaptive Channel Pruning and Effective Weights Inheritance
• ABCP: Automatic Block-wise and Channel-wise Network Pruning via Joint Search
• Channel Pruning via Multi-Criteria based on Weight Dependency
• Pruning with Compensation: Efficient Channel Pruning for Deep Convolutional Neural Networks
• Localizing Interpretable Multi-scale informative Patches Derived from Media Classification Task
• Prioritized Subnet Sampling for Resource-Adaptive Supernet Training
• Effective Model Compression via Stage-wise Pruning
• Differentiable Network Adaption with Elastic Search Space
• Blending Pruning Criteria for Convolutional Neural Networks