Network Pruning via Transformable Architecture Search

Abstract

Network pruning reduces the computation costs of an over-parameterized network without performance damage. Prevailing pruning algorithms pre-define the width and depth of the pruned networks, and then transfer parameters from the unpruned network to pruned networks. To break the structure limitation of the pruned networks, we propose to apply neural architecture search to search directly for a network with flexible channel and layer sizes. The number of the channels/layers is learned by minimizing the loss of the pruned networks. The feature map of the pruned network is an aggregation of K feature map fragments (generated by K networks of different sizes), which are sampled based on the probability distribution.The loss can be back-propagated not only to the network weights, but also to the parameterized distribution to explicitly tune the size of the channels/layers. Specifically, we apply channel-wise interpolation to keep the feature map with different channel sizes aligned in the aggregation procedure. The maximum probability for the size in each distribution serves as the width and depth of the pruned network, whose parameters are learned by knowledge transfer, e.g., knowledge distillation, from the original networks. Experiments on CIFAR-10, CIFAR-100 and ImageNet demonstrate the effectiveness of our new perspective of network pruning compared to traditional network pruning algorithms. Various searching and knowledge transfer approaches are conducted to show the effectiveness of the two components. Code is at: https://github.com/D-X-Y/NAS-Projects.

arXiv commentPublished in the 33rd Conference on Neural Information Processing Systems (NeurIPS 2019)

References in corpus (14)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Distilling the Knowledge in a Neural Network
• Neural Architecture Search with Reinforcement Learning
• Categorical Reparameterization with Gumbel-Softmax
• MobileNetV2: Inverted Residuals and Linear Bottlenecks
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• DARTS: Differentiable Architecture Search
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Rethinking the Value of Network Pruning
• Efficient Architecture Search by Network Transformation
• Learning the Number of Neurons in Deep Networks
• AutoSlim: Towards One-Shot Architecture Search for Channel Numbers

Cited by in corpus (19)

• AutoML: A Survey of the State-of-the-Art
• One-Shot Neural Architecture Search via Self-Evaluated Template Network
• NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size
• HRel: Filter Pruning based on High Relevance between Activation Maps and Class Labels
• DAIS: Automatic Channel Pruning via Differentiable Annealing Indicator Search
• Knowledge Distillation Methods for Efficient Unsupervised Adaptation Across Multiple Domains
• FreezeNet: Full Performance by Reduced Storage Costs
• DASS: Differentiable Architecture Search for Sparse neural networks
• Densely Connected Search Space for More Flexible Neural Architecture Search
• Exploring Gradient Flow Based Saliency for DNN Model Compression
• Teacher Supervises Students How to Learn From Partially Labeled Images for Facial Landmark Detection
• AMEIR: Automatic Behavior Modeling, Interaction Exploration and MLP Investigation in the Recommender System
• RARTS: An Efficient First-Order Relaxed Architecture Search Method
• Towards Efficient Convolutional Network Models with Filter Distribution Templates
• Network Adjustment: Channel Search Guided by FLOPs Utilization Ratio
• Self-Constructing Neural Networks Through Random Mutation
• Differentiable Network Adaption with Elastic Search Space
• Reconsidering CO2 emissions from Computer Vision
• Differentiable Neural Architecture Search with Morphism-based Transformable Backbone Architectures