Few-shot Neural Architecture Search
arXiv:2006.06863
Abstract
Efficient evaluation of a network architecture drawn from a large search space remains a key challenge in Neural Architecture Search (NAS). Vanilla NAS evaluates each architecture by training from scratch, which gives the true performance but is extremely time-consuming. Recently, one-shot NAS substantially reduces the computation cost by training only one supernetwork, a.k.a. supernet, to approximate the performance of every architecture in the search space via weight-sharing. However, the performance estimation can be very inaccurate due to the co-adaption among operations. In this paper, we propose few-shot NAS that uses multiple supernetworks, called sub-supernet, each covering different regions of the search space to alleviate the undesired co-adaption. Compared to one-shot NAS, few-shot NAS improves the accuracy of architecture evaluation with a small increase of evaluation cost. With only up to 7 sub-supernets, few-shot NAS establishes new SoTAs: on ImageNet, it finds models that reach 80.5% top-1 accuracy at 600 MB FLOPS and 77.5% top-1 accuracy at 238 MFLOPS; on CIFAR10, it reaches 98.72% top-1 accuracy without using extra data or transfer learning. In Auto-GAN, few-shot NAS outperforms the previously published results by up to 20%. Extensive experiments show that few-shot NAS significantly improves various one-shot methods, including 4 gradient-based and 6 search-based methods on 3 different tasks in NasBench-201 and NasBench1-shot-1.
References in corpus (23)
- Spectral Normalization for Generative Adversarial Networks
- Improved Regularization of Convolutional Neural Networks with Cutout
- DARTS: Differentiable Architecture Search
- AutoAugment: Learning Augmentation Policies from Data
- Neural Architecture Optimization
- Designing Neural Network Architectures using Reinforcement Learning
- PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
- Understanding and Robustifying Differentiable Architecture Search
- NAS-Bench-101: Towards Reproducible Neural Architecture Search
- Single Path One-Shot Neural Architecture Search with Uniform Sampling
- One-Shot Neural Architecture Search via Self-Evaluated Template Network
- DetNAS: Backbone Search for Object Detection
- NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search
- Theoretical Analysis of Auto Rate-Tuning by Batch Normalization
- AlphaX: eXploring Neural Architectures with Deep Neural Networks and Monte Carlo Tree Search
- Adaptive Stochastic Natural Gradient Method for One-Shot Neural Architecture Search
- AtomNAS: Fine-Grained End-to-End Neural Architecture Search
- Sample-Efficient Neural Architecture Search by Learning Action Space
- FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensions
- How to Train Your Super-Net: An Analysis of Training Heuristics in Weight-Sharing NAS
- Progressive DARTS: Bridging the Optimization Gap for NAS in the Wild
- Angle-based Search Space Shrinking for Neural Architecture Search
- Balanced One-shot Neural Architecture Optimization