Bridging the Gap between Sample-based and One-shot Neural Architecture Search with BONAS

Abstract

Neural Architecture Search (NAS) has shown great potentials in finding better neural network designs. Sample-based NAS is the most reliable approach which aims at exploring the search space and evaluating the most promising architectures. However, it is computationally very costly. As a remedy, the one-shot approach has emerged as a popular technique for accelerating NAS using weight-sharing. However, due to the weight-sharing of vastly different networks, the one-shot approach is less reliable than the sample-based approach. In this work, we propose BONAS (Bayesian Optimized Neural Architecture Search), a sample-based NAS framework which is accelerated using weight-sharing to evaluate multiple related architectures simultaneously. Specifically, we apply Graph Convolutional Network predictor as a surrogate model for Bayesian Optimization to select multiple related candidate models in each iteration. We then apply weight-sharing to train multiple candidate models simultaneously. This approach not only accelerates the traditional sample-based approach significantly, but also keeps its reliability. This is because weight-sharing among related architectures are more reliable than those in the one-shot approach. Extensive experiments are conducted to verify the effectiveness of our method over many competing algorithms.

Accepted by NeurIPS 2020

References in corpus (17)

• Practical Bayesian Optimization of Machine Learning Algorithms
• Neural Architecture Search with Reinforcement Learning
• Improved Regularization of Convolutional Neural Networks with Cutout
• DARTS: Differentiable Architecture Search
• Neural Architecture Optimization
• Scalable Bayesian Optimization Using Deep Neural Networks
• NAS-Bench-101: Towards Reproducible Neural Architecture Search
• The Evolved Transformer
• Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling
• Evaluating the Search Phase of Neural Architecture Search
• NAS evaluation is frustratingly hard
• BayesNAS: A Bayesian Approach for Neural Architecture Search
• Fast, Accurate and Lightweight Super-Resolution with Neural Architecture Search
• AlphaX: eXploring Neural Architectures with Deep Neural Networks and Monte Carlo Tree Search
• Adaptive Stochastic Natural Gradient Method for One-Shot Neural Architecture Search
• Sample-Efficient Neural Architecture Search by Learning Action Space
• Partial Order Pruning: for Best Speed/Accuracy Trade-off in Neural Architecture Search

Cited by in corpus (18)

• BRP-NAS: Prediction-based NAS using GCNs
• Automated Machine Learning on Graphs: A Survey
• Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?
• Sample-Efficient Neural Architecture Search by Learning Action Space
• Evaluating Efficient Performance Estimators of Neural Architectures
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• One Proxy Device Is Enough for Hardware-Aware Neural Architecture Search
• A Generic Graph-based Neural Architecture Encoding Scheme for Predictor-based NAS
• How Powerful are Performance Predictors in Neural Architecture Search?
• Stronger NAS with Weaker Predictors
• CATE: Computation-aware Neural Architecture Encoding with Transformers
• Fitting the Search Space of Weight-sharing NAS with Graph Convolutional Networks
• FNAS: Uncertainty-Aware Fast Neural Architecture Search
• A Study on Encodings for Neural Architecture Search
• Heed the Noise in Performance Evaluations in Neural Architecture Search
• Joint-DetNAS: Upgrade Your Detector with NAS, Pruning and Dynamic Distillation
• Scaling Up Deep Neural Network Optimization for Edge Inference
• Training BatchNorm Only in Neural Architecture Search and Beyond