Rapid Elastic Architecture Search under Specialized Classes and Resource Constraints

Abstract

In many real-world applications, we often need to handle various deployment scenarios, where the resource constraint and the superclass of interest corresponding to a group of classes are dynamically specified. How to efficiently deploy deep models for diverse deployment scenarios is a new challenge. Previous NAS approaches seek to design architectures for all classes simultaneously, which may not be optimal for some individual superclasses. A straightforward solution is to search an architecture from scratch for each deployment scenario, which however is computation-intensive and impractical. To address this, we present a novel and general framework, called Elastic Architecture Search (EAS), permitting instant specializations at runtime for diverse superclasses with various resource constraints. To this end, we first propose to effectively train an over-parameterized network via a superclass dropout strategy during training. In this way, the resulting model is robust to the subsequent superclasses dropping at inference time. Based on the well-trained over-parameterized network, we then propose an efficient architecture generator to obtain promising architectures within a single forward pass. Experiments on three image classification datasets show that EAS is able to find more compact networks with better performance while remarkably being orders of magnitude faster than state-of-the-art NAS methods, e.g., outperforming OFA (once-for-all) by 1.3% on Top-1 accuracy at a budget around 361M #MAdds on ImageNet-10. More critically, EAS is able to find compact architectures within 0.1 second for 50 deployment scenarios.

arXiv commentTech report

References in corpus (24)

• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Improving neural networks by preventing co-adaptation of feature detectors
• Neural Architecture Search with Reinforcement Learning
• Categorical Reparameterization with Gumbel-Softmax
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• DARTS: Differentiable Architecture Search
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Rethinking the Value of Network Pruning
• Large-Scale Evolution of Image Classifiers
• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• Multi-Scale Dense Networks for Resource Efficient Image Classification
• Discrimination-aware Network Pruning for Deep Model Compression
• AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning
• NAT: Neural Architecture Transformer for Accurate and Compact Architectures
• Breaking the Curse of Space Explosion: Towards Efficient NAS with Curriculum Search
• LiteEval: A Coarse-to-Fine Framework for Resource Efficient Video Recognition
• Neural Architecture Generator Optimization
• Learning to Zoom: a Saliency-Based Sampling Layer for Neural Networks
• Pareto-Frontier-aware Neural Architecture Generation for Diverse Budgets

Cited by in corpus (1)

• Pruning Self-attentions into Convolutional Layers in Single Path