How Powerful are Performance Predictors in Neural Architecture Search?

Abstract

Early methods in the rapidly developing field of neural architecture search (NAS) required fully training thousands of neural networks. To reduce this extreme computational cost, dozens of techniques have since been proposed to predict the final performance of neural architectures. Despite the success of such performance prediction methods, it is not well-understood how different families of techniques compare to one another, due to the lack of an agreed-upon evaluation metric and optimization for different constraints on the initialization time and query time. In this work, we give the first large-scale study of performance predictors by analyzing 31 techniques ranging from learning curve extrapolation, to weight-sharing, to supervised learning, to "zero-cost" proxies. We test a number of correlation- and rank-based performance measures in a variety of settings, as well as the ability of each technique to speed up predictor-based NAS frameworks. Our results act as recommendations for the best predictors to use in different settings, and we show that certain families of predictors can be combined to achieve even better predictive power, opening up promising research directions. Our code, featuring a library of 31 performance predictors, is available at https://github.com/automl/naslib.

arXiv commentNeurIPS 2021

References in corpus (27)

• Neural Architecture Search with Reinforcement Learning
• Neural Architecture Search: A Survey
• DARTS: Differentiable Architecture Search
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• Neural Architecture Optimization
• Scalable Bayesian Optimization Using Deep Neural Networks
• Understanding and Robustifying Differentiable Architecture Search
• DARTS+: Improved Differentiable Architecture Search with Early Stopping
• Random Search and Reproducibility for Neural Architecture Search
• Evaluating the Search Phase of Neural Architecture Search
• Picking Winning Tickets Before Training by Preserving Gradient Flow
• Carbon Emissions and Large Neural Network Training
• Faster gaze prediction with dense networks and Fisher pruning
• NAS evaluation is frustratingly hard
• Best Practices for Scientific Research on Neural Architecture Search
• NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search
• Bridging the Gap between Sample-based and One-shot Neural Architecture Search with BONAS
• AlphaX: eXploring Neural Architectures with Deep Neural Networks and Monte Carlo Tree Search
• Raiders of the Lost Architecture: Kernels for Bayesian Optimization in Conditional Parameter Spaces
• Semi-Supervised Neural Architecture Search
• Neural Architecture Search without Training
• Deeper Insights into Weight Sharing in Neural Architecture Search
• A Generic Graph-based Neural Architecture Encoding Scheme for Predictor-based NAS
• How to Train Your Super-Net: An Analysis of Training Heuristics in Weight-Sharing NAS
• Evolutionary-Neural Hybrid Agents for Architecture Search
• BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search
• A Novel Training Protocol for Performance Predictors of Evolutionary Neural Architecture Search Algorithms

Cited by in corpus (7)

• Evaluating Efficient Performance Estimators of Neural Architectures
• Speedy Performance Estimation for Neural Architecture Search
• Exploring the Loss Landscape in Neural Architecture Search
• Evolutionary Neural Cascade Search across Supernetworks
• Training BatchNorm Only in Neural Architecture Search and Beyond
• Mutation is all you need
• Guided Evolution for Neural Architecture Search