Once-for-All: Train One Network and Specialize it for Efficient Deployment

Abstract

We address the challenging problem of efficient inference across many devices and resource constraints, especially on edge devices. Conventional approaches either manually design or use neural architecture search (NAS) to find a specialized neural network and train it from scratch for each case, which is computationally prohibitive (causing $C{O}_2$ emission as much as 5 cars' lifetime) thus unscalable. In this work, we propose to train a once-for-all (OFA) network that supports diverse architectural settings by decoupling training and search, to reduce the cost. We can quickly get a specialized sub-network by selecting from the OFA network without additional training. To efficiently train OFA networks, we also propose a novel progressive shrinking algorithm, a generalized pruning method that reduces the model size across many more dimensions than pruning (depth, width, kernel size, and resolution). It can obtain a surprisingly large number of sub-networks ($>10^{19}$) that can fit different hardware platforms and latency constraints while maintaining the same level of accuracy as training independently. On diverse edge devices, OFA consistently outperforms state-of-the-art (SOTA) NAS methods (up to 4.0% ImageNet top1 accuracy improvement over MobileNetV3, or same accuracy but 1.5x faster than MobileNetV3, 2.6x faster than EfficientNet w.r.t measured latency) while reducing many orders of magnitude GPU hours and $C{O}_2$ emission. In particular, OFA achieves a new SOTA 80.0% ImageNet top-1 accuracy under the mobile setting ($<$600M MACs). OFA is the winning solution for the 3rd Low Power Computer Vision Challenge (LPCVC), DSP classification track and the 4th LPCVC, both classification track and detection track. Code and 50 pre-trained models (for many devices & many latency constraints) are released at https://github.com/mit-han-lab/once-for-all.

ICLR 2020

References in corpus (13)

• Distilling the Knowledge in a Neural Network
• Neural Architecture Search with Reinforcement Learning
• MobileNetV2: Inverted Residuals and Linear Bottlenecks
• DARTS: Differentiable Architecture Search
• Trained Ternary Quantization
• Efficient Architecture Search by Network Transformation
• Learning Efficient Convolutional Networks through Network Slimming
• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• Path-Level Network Transformation for Efficient Architecture Search
• AutoSlim: Towards One-Shot Architecture Search for Channel Numbers
• N2N Learning: Network to Network Compression via Policy Gradient Reinforcement Learning
• Superposition of many models into one
• Stochastic Downsampling for Cost-Adjustable Inference and Improved Regularization in Convolutional Networks

Cited by in corpus (18)

• Empowering Edge Intelligence: A Comprehensive Survey on On-Device AI Models
• Review: Deep Learning in Electron Microscopy
• Compute and memory efficient universal sound source separation
• One Proxy Device Is Enough for Hardware-Aware Neural Architecture Search
• AdaSpring: Context-adaptive and Runtime-evolutionary Deep Model Compression for Mobile Applications
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• Anytime Inference with Distilled Hierarchical Neural Ensembles
• Evolutionary Spiking Neural Networks: A Survey
• DASS: Differentiable Architecture Search for Sparse neural networks
• SalNAS: Efficient Saliency-prediction Neural Architecture Search with self-knowledge distillation
• Greening AI-enabled Systems with Software Engineering: A Research Agenda for Environmentally Sustainable AI Practices
• S3NAS: Fast NPU-aware Neural Architecture Search Methodology
• ModuleNet: Knowledge-inherited Neural Architecture Search
• GENNAPE: Towards Generalized Neural Architecture Performance Estimators
• The Benefit of the Doubt: Uncertainty Aware Sensing for Edge Computing Platforms
• AIO-P: Expanding Neural Performance Predictors Beyond Image Classification
• XploreNAS: Explore Adversarially Robust & Hardware-efficient Neural Architectures for Non-ideal Xbars
• One-Shot Neural Ensemble Architecture Search by Diversity-Guided Search Space Shrinking