Fitting the Search Space of Weight-sharing NAS with Graph Convolutional Networks
arXiv:2004.08423
Abstract
Neural architecture search has attracted wide attentions in both academia and industry. To accelerate it, researchers proposed weight-sharing methods which first train a super-network to reuse computation among different operators, from which exponentially many sub-networks can be sampled and efficiently evaluated. These methods enjoy great advantages in terms of computational costs, but the sampled sub-networks are not guaranteed to be estimated precisely unless an individual training process is taken. This paper owes such inaccuracy to the inevitable mismatch between assembled network layers, so that there is a random error term added to each estimation. We alleviate this issue by training a graph convolutional network to fit the performance of sampled sub-networks so that the impact of random errors becomes minimal. With this strategy, we achieve a higher rank correlation coefficient in the selected set of candidates, which consequently leads to better performance of the final architecture. In addition, our approach also enjoys the flexibility of being used under different hardware constraints, since the graph convolutional network has provided an efficient lookup table of the performance of architectures in the entire search space.
arXiv commentAccepted to AAAI 2021
References in corpus (19)
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Neural Architecture Search with Reinforcement Learning
- Neural Architecture Search: A Survey
- Large-Scale Evolution of Image Classifiers
- Neural Architecture Optimization
- PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
- SMASH: One-Shot Model Architecture Search through HyperNetworks
- Efficient Architecture Search by Network Transformation
- SNAS: Stochastic Neural Architecture Search
- NAS-Bench-101: Towards Reproducible Neural Architecture Search
- Single Path One-Shot Neural Architecture Search with Uniform Sampling
- Fast, Accurate and Lightweight Super-Resolution with Neural Architecture Search
- NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search
- Bridging the Gap between Sample-based and One-shot Neural Architecture Search with BONAS
- GOLD-NAS: Gradual, One-Level, Differentiable
- Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
- A Generic Graph-based Neural Architecture Encoding Scheme for Predictor-based NAS
- Densely Connected Search Space for More Flexible Neural Architecture Search
- A Variational-Sequential Graph Autoencoder for Neural Architecture Performance Prediction