BS-NAS: Broadening-and-Shrinking One-Shot NAS with Searchable Numbers of Channels
arXiv:2003.09821
Abstract
One-Shot methods have evolved into one of the most popular methods in Neural Architecture Search (NAS) due to weight sharing and single training of a supernet. However, existing methods generally suffer from two issues: predetermined number of channels in each layer which is suboptimal; and model averaging effects and poor ranking correlation caused by weight coupling and continuously expanding search space. To explicitly address these issues, in this paper, a Broadening-and-Shrinking One-Shot NAS (BS-NAS) framework is proposed, in which `broadening' refers to broadening the search space with a spring block enabling search for numbers of channels during training of the supernet; while `shrinking' refers to a novel shrinking strategy gradually turning off those underperforming operations. The above innovations broaden the search space for wider representation and then shrink it by gradually removing underperforming operations, followed by an evolutionary algorithm to efficiently search for the optimal architecture. Extensive experiments on ImageNet illustrate the effectiveness of the proposed BS-NAS as well as the state-of-the-art performance.
arXiv comment14 pages
References in corpus (12)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- mixup: Beyond Empirical Risk Minimization
- Neural Architecture Search with Reinforcement Learning
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- SGDR: Stochastic Gradient Descent with Warm Restarts
- AMC: AutoML for Model Compression and Acceleration on Mobile Devices
- Large-Scale Evolution of Image Classifiers
- MixConv: Mixed Depthwise Convolutional Kernels
- Single Path One-Shot Neural Architecture Search with Uniform Sampling
- Understanding Neural Architecture Search Techniques
- Improving One-shot NAS by Suppressing the Posterior Fading