Speedy Performance Estimation for Neural Architecture Search
arXiv:2006.04492
Abstract
Reliable yet efficient evaluation of generalisation performance of a proposed architecture is crucial to the success of neural architecture search (NAS). Traditional approaches face a variety of limitations: training each architecture to completion is prohibitively expensive, early stopped validation accuracy may correlate poorly with fully trained performance, and model-based estimators require large training sets. We instead propose to estimate the final test performance based on a simple measure of training speed. Our estimator is theoretically motivated by the connection between generalisation and training speed, and is also inspired by the reformulation of a PAC-Bayes bound under the Bayesian setting. Our model-free estimator is simple, efficient, and cheap to implement, and does not require hyperparameter-tuning or surrogate training before deployment. We demonstrate on various NAS search spaces that our estimator consistently outperforms other alternatives in achieving better correlation with the true test performance rankings. We further show that our estimator can be easily incorporated into both query-based and one-shot NAS methods to improve the speed or quality of the search.
arXiv comment23 pages, 14 figures
References in corpus (22)
- Neural Architecture Search with Reinforcement Learning
- Neural Architecture Search: A Survey
- DARTS: Differentiable Architecture Search
- Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization
- Large-Scale Evolution of Image Classifiers
- Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
- NAS-Bench-101: Towards Reproducible Neural Architecture Search
- Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks
- Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets
- NAS evaluation is frustratingly hard
- Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search
- Calibrating Deep Neural Networks using Focal Loss
- Exploring Randomly Wired Neural Networks for Image Recognition
- NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search
- The large learning rate phase of deep learning: the catapult mechanism
- Stiffness: A New Perspective on Generalization in Neural Networks
- Algorithmic stability and hypothesis complexity
- Neural Architecture Search without Training
- Information-Theoretic Generalization Bounds for SGLD via Data-Dependent Estimates
- On the Origin of Implicit Regularization in Stochastic Gradient Descent
- Meta Architecture Search
- How Powerful are Performance Predictors in Neural Architecture Search?