Rapid Elastic Architecture Search under Specialized Classes and Resource Constraints

Abstract

In many real-world applications, we often need to handle various deployment scenarios, where the resource constraint and the superclass of interest corresponding to a group of classes are dynamically specified. How to efficiently deploy deep models for diverse deployment scenarios is a new challenge. Previous NAS approaches seek to design architectures for all classes simultaneously, which may not be optimal for some individual superclasses. A straightforward solution is to search an architecture from scratch for each deployment scenario, which however is computation-intensive and impractical. To address this, we present a novel and general framework, called Elastic Architecture Search (EAS), permitting instant specializations at runtime for diverse superclasses with various resource constraints. To this end, we first propose to effectively train an over-parameterized network via a superclass dropout strategy during training. In this way, the resulting model is robust to the subsequent superclasses dropping at inference time. Based on the well-trained over-parameterized network, we then propose an efficient architecture generator to obtain promising architectures within a single forward pass. Experiments on three image classification datasets show that EAS is able to find more compact networks with better performance while remarkably being orders of magnitude faster than state-of-the-art NAS methods, e.g., outperforming OFA (once-for-all) by 1.3% on Top-1 accuracy at a budget around 361M #MAdds on ImageNet-10. More critically, EAS is able to find compact architectures within 0.1 second for 50 deployment scenarios.

Tech report

References in corpus (9)

• Distilling the Knowledge in a Neural Network
• Improving neural networks by preventing co-adaptation of feature detectors
• Neural Architecture Search with Reinforcement Learning
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• NAT: Neural Architecture Transformer for Accurate and Compact Architectures
• Breaking the Curse of Space Explosion: Towards Efficient NAS with Curriculum Search
• LiteEval: A Coarse-to-Fine Framework for Resource Efficient Video Recognition
• Pareto-Frontier-aware Neural Architecture Generation for Diverse Budgets

Cited by in corpus (1)

• Pruning Self-attentions into Convolutional Layers in Single Path