AutoSlim: Towards One-Shot Architecture Search for Channel Numbers

Abstract

We study how to set channel numbers in a neural network to achieve better accuracy under constrained resources (e.g., FLOPs, latency, memory footprint or model size). A simple and one-shot solution, named AutoSlim, is presented. Instead of training many network samples and searching with reinforcement learning, we train a single slimmable network to approximate the network accuracy of different channel configurations. We then iteratively evaluate the trained slimmable model and greedily slim the layer with minimal accuracy drop. By this single pass, we can obtain the optimized channel configurations under different resource constraints. We present experiments with MobileNet v1, MobileNet v2, ResNet-50 and RL-searched MNasNet on ImageNet classification. We show significant improvements over their default channel configurations. We also achieve better accuracy than recent channel pruning methods and neural architecture search methods. Notably, by setting optimized channel numbers, our AutoSlim-MobileNet-v2 at 305M FLOPs achieves 74.2% top-1 accuracy, 2.4% better than default MobileNet-v2 (301M FLOPs), and even 0.2% better than RL-searched MNasNet (317M FLOPs). Our AutoSlim-ResNet-50 at 570M FLOPs, without depthwise convolutions, achieves 1.3% better accuracy than MobileNet-v1 (569M FLOPs). Code and models will be available at: https://github.com/JiahuiYu/slimmable_networks

arXiv commenttech report

References in corpus (19)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Continuous control with deep reinforcement learning
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Neural Architecture Search with Reinforcement Learning
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Neural Architecture Search: A Survey
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• Pruning Filters for Efficient ConvNets
• Emergence of Locomotion Behaviours in Rich Environments
• Efficient Neural Architecture Search via Parameter Sharing
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• Neural Architecture Optimization
• SMASH: One-Shot Model Architecture Search through HyperNetworks
• Wide Activation for Efficient and Accurate Image Super-Resolution
• Slimmable Neural Networks
• Hierarchical Representations for Efficient Architecture Search
• Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers
• Graph HyperNetworks for Neural Architecture Search
• SkipNet: Learning Dynamic Routing in Convolutional Networks

Cited by in corpus (53)

• Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications
• Network Pruning via Transformable Architecture Search
• Optimization for deep learning: theory and algorithms
• DynaBERT: Dynamic BERT with Adaptive Width and Depth
• Soft Threshold Weight Reparameterization for Learnable Sparsity
• Once-for-All: Train One Network and Specialize it for Efficient Deployment
• HRel: Filter Pruning based on High Relevance between Activation Maps and Class Labels
• BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models
• Contrastive Self-supervised Neural Architecture Search
• Searching for Low-Bit Weights in Quantized Neural Networks
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• On the Benefit of Width for Neural Networks: Disappearance of Bad Basins
• CARS: Continuous Evolution for Efficient Neural Architecture Search
• Dual-mode ASR: Unify and Improve Streaming ASR with Full-context Modeling
• Locally Free Weight Sharing for Network Width Search
• AdaSpring: Context-adaptive and Runtime-evolutionary Deep Model Compression for Mobile Applications
• Switchable Precision Neural Networks
• Pruning Self-attentions into Convolutional Layers in Single Path
• DMCP: Differentiable Markov Channel Pruning for Neural Networks
• Pruning via Iterative Ranking of Sensitivity Statistics
• EagleEye: Fast Sub-net Evaluation for Efficient Neural Network Pruning
• Searching for Efficient Multi-Stage Vision Transformers
• An Information Theory-inspired Strategy for Automatic Network Pruning
• Cascaded channel pruning using hierarchical self-distillation
• An Image Enhancing Pattern-based Sparsity for Real-time Inference on Mobile Devices
• K-shot NAS: Learnable Weight-Sharing for NAS with K-shot Supernets
• Filter Sketch for Network Pruning
• Towards Learning of Filter-Level Heterogeneous Compression of Convolutional Neural Networks
• Towards Backward-Compatible Representation Learning
• Dynamic-OFA: Runtime DNN Architecture Switching for Performance Scaling on Heterogeneous Embedded Platforms
• AACP: Model Compression by Accurate and Automatic Channel Pruning
• DS-Net++: Dynamic Weight Slicing for Efficient Inference in CNNs and Transformers
• NetAdaptV2: Efficient Neural Architecture Search with Fast Super-Network Training and Architecture Optimization
• HALP: Hardware-Aware Latency Pruning
• Recursive-NeRF: An Efficient and Dynamically Growing NeRF
• MemNet: Memory-Efficiency Guided Neural Architecture Search with Augment-Trim learning
• Joint Channel and Weight Pruning for Model Acceleration on Moblie Devices
• Slimmable Generative Adversarial Networks
• Towards Efficient Convolutional Network Models with Filter Distribution Templates
• BCNet: Searching for Network Width with Bilaterally Coupled Network
• AdaPruner: Adaptive Channel Pruning and Effective Weights Inheritance
• Dynamic Sparsity Neural Networks for Automatic Speech Recognition
• The Heterogeneity Hypothesis: Finding Layer-Wise Differentiated Network Architectures
• Dynamic Slimmable Denoising Network
• RANP: Resource Aware Neuron Pruning at Initialization for 3D CNNs
• Neural Inheritance Relation Guided One-Shot Layer Assignment Search
• Out-of-the-box channel pruned networks
• Effective Model Compression via Stage-wise Pruning
• Width Transfer: On the (In)variance of Width Optimization
• Masked Training of Neural Networks with Partial Gradients
• Network Adjustment: Channel Search Guided by FLOPs Utilization Ratio
• Prioritized Subnet Sampling for Resource-Adaptive Supernet Training
• Joslim: Joint Widths and Weights Optimization for Slimmable Neural Networks