BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models

Abstract

Neural architecture search (NAS) has shown promising results discovering models that are both accurate and fast. For NAS, training a one-shot model has become a popular strategy to rank the relative quality of different architectures (child models) using a single set of shared weights. However, while one-shot model weights can effectively rank different network architectures, the absolute accuracies from these shared weights are typically far below those obtained from stand-alone training. To compensate, existing methods assume that the weights must be retrained, finetuned, or otherwise post-processed after the search is completed. These steps significantly increase the compute requirements and complexity of the architecture search and model deployment. In this work, we propose BigNAS, an approach that challenges the conventional wisdom that post-processing of the weights is necessary to get good prediction accuracies. Without extra retraining or post-processing steps, we are able to train a single set of shared weights on ImageNet and use these weights to obtain child models whose sizes range from 200 to 1000 MFLOPs. Our discovered model family, BigNASModels, achieve top-1 accuracies ranging from 76.5% to 80.9%, surpassing state-of-the-art models in this range including EfficientNets and Once-for-All networks without extra retraining or post-processing. We present ablative study and analysis to further understand the proposed BigNASModels.

arXiv commentAccepted in ECCV 2020

References in corpus (10)

• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Neural Architecture Search with Reinforcement Learning
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Searching for Activation Functions
• AutoAugment: Learning Augmentation Policies from Data
• Efficient Neural Architecture Search via Parameter Sharing
• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• AutoSlim: Towards One-Shot Architecture Search for Channel Numbers

Cited by in corpus (23)

• AutoML: A Survey of the State-of-the-Art
• Cream of the Crop: Distilling Prioritized Paths For One-Shot Neural Architecture Search
• Searching for Low-Bit Weights in Quantized Neural Networks
• TinyTL: Reduce Activations, Not Trainable Parameters for Efficient On-Device Learning
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• FBNetV3: Joint Architecture-Recipe Search using Predictor Pretraining
• Anytime Inference with Distilled Hierarchical Neural Ensembles
• Analyzing and Mitigating Interference in Neural Architecture Search
• AttentiveNAS: Improving Neural Architecture Search via Attentive Sampling
• You Only Compress Once: Towards Effective and Elastic BERT Compression via Exploit-Explore Stochastic Nature Gradient
• A3D: Adaptive 3D Networks for Video Action Recognition
• Understanding and Accelerating Neural Architecture Search with Training-Free and Theory-Grounded Metrics
• ScaleNAS: One-Shot Learning of Scale-Aware Representations for Visual Recognition
• Parameter Prediction for Unseen Deep Architectures
• ViPNAS: Efficient Video Pose Estimation via Neural Architecture Search
• SuperShaper: Task-Agnostic Super Pre-training of BERT Models with Variable Hidden Dimensions
• Dynamic Sparsity Neural Networks for Automatic Speech Recognition
• Auto-NBA: Efficient and Effective Search Over the Joint Space of Networks, Bitwidths, and Accelerators
• FP-NAS: Fast Probabilistic Neural Architecture Search
• Efficient Sampling for Predictor-Based Neural Architecture Search
• An Efficient Quantitative Approach for Optimizing Convolutional Neural Networks
• One-Shot Neural Ensemble Architecture Search by Diversity-Guided Search Space Shrinking
• Mutually-aware Sub-Graphs Differentiable Architecture Search