Neural Predictor for Neural Architecture Search
arXiv:1912.00848
Abstract
Neural Architecture Search methods are effective but often use complex algorithms to come up with the best architecture. We propose an approach with three basic steps that is conceptually much simpler. First we train N random architectures to generate N (architecture, validation accuracy) pairs and use them to train a regression model that predicts accuracy based on the architecture. Next, we use this regression model to predict the validation accuracies of a large number of random architectures. Finally, we train the top-K predicted architectures and deploy the model with the best validation result. While this approach seems simple, it is more than 20 times as sample efficient as Regularized Evolution on the NASBench-101 benchmark and can compete on ImageNet with more complex approaches based on weight sharing, such as ProxylessNAS.
References in corpus (11)
- Adam: A Method for Stochastic Optimization
- Semi-Supervised Classification with Graph Convolutional Networks
- SGDR: Stochastic Gradient Descent with Warm Restarts
- Efficient Neural Architecture Search via Parameter Sharing
- Large-Scale Evolution of Image Classifiers
- Neural Architecture Optimization
- SMASH: One-Shot Model Architecture Search through HyperNetworks
- SNAS: Stochastic Neural Architecture Search
- NAS-Bench-101: Towards Reproducible Neural Architecture Search
- Random Search and Reproducibility for Neural Architecture Search
- Efficient Forward Architecture Search
Cited by in corpus (15)
- BRP-NAS: Prediction-based NAS using GCNs
- Zero-Cost Proxies for Lightweight NAS
- Semi-Supervised Neural Architecture Search
- A Learned Performance Model for Tensor Processing Units
- Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?
- FBNetV3: Joint Architecture-Recipe Search using Predictor Pretraining
- Stronger NAS with Weaker Predictors
- NASGEM: Neural Architecture Search via Graph Embedding Method
- FNAS: Uncertainty-Aware Fast Neural Architecture Search
- LightSpeech: Lightweight and Fast Text to Speech with Neural Architecture Search
- A Study on Encodings for Neural Architecture Search
- Parameter Prediction for Unseen Deep Architectures
- Efficient Sampling for Predictor-Based Neural Architecture Search
- Zero-Cost Operation Scoring in Differentiable Architecture Search
- Pretraining Neural Architecture Search Controllers with Locality-based Self-Supervised Learning