Rethinking Performance Estimation in Neural Architecture Search

Abstract

Neural architecture search (NAS) remains a challenging problem, which is attributed to the indispensable and time-consuming component of performance estimation (PE). In this paper, we provide a novel yet systematic rethinking of PE in a resource constrained regime, termed budgeted PE (BPE), which precisely and effectively estimates the performance of an architecture sampled from an architecture space. Since searching an optimal BPE is extremely time-consuming as it requires to train a large number of networks for evaluation, we propose a Minimum Importance Pruning (MIP) approach. Given a dataset and a BPE search space, MIP estimates the importance of hyper-parameters using random forest and subsequently prunes the minimum one from the next iteration. In this way, MIP effectively prunes less important hyper-parameters to allocate more computational resource on more important ones, thus achieving an effective exploration. By combining BPE with various search algorithms including reinforcement learning, evolution algorithm, random search, and differentiable architecture search, we achieve 1, 000x of NAS speed up with a negligible performance drop comparing to the SOTA

References in corpus (14)

• Practical Bayesian Optimization of Machine Learning Algorithms
• Neural Architecture Search with Reinforcement Learning
• End to End Learning for Self-Driving Cars
• Improved Regularization of Convolutional Neural Networks with Cutout
• DARTS: Differentiable Architecture Search
• Searching for Efficient Multi-Scale Architectures for Dense Image Prediction
• Efficient Architecture Search by Network Transformation
• FreeAnchor: Learning to Match Anchors for Visual Object Detection
• Random Search and Reproducibility for Neural Architecture Search
• Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation
• Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets
• Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search
• Adaptive Stochastic Natural Gradient Method for One-Shot Neural Architecture Search
• Binarized Neural Architecture Search

Cited by in corpus (1)

• AgEBO-Tabular: Joint Neural Architecture and Hyperparameter Search with Autotuned Data-Parallel Training for Tabular Data