Neural Architecture Search without Training

Abstract

The time and effort involved in hand-designing deep neural networks is immense. This has prompted the development of Neural Architecture Search (NAS) techniques to automate this design. However, NAS algorithms tend to be slow and expensive; they need to train vast numbers of candidate networks to inform the search process. This could be alleviated if we could partially predict a network's trained accuracy from its initial state. In this work, we examine the overlap of activations between datapoints in untrained networks and motivate how this can give a measure which is usefully indicative of a network's trained performance. We incorporate this measure into a simple algorithm that allows us to search for powerful networks without any training in a matter of seconds on a single GPU, and verify its effectiveness on NAS-Bench-101, NAS-Bench-201, NATS-Bench, and Network Design Spaces. Our approach can be readily combined with more expensive search methods; we examine a simple adaptation of regularised evolutionary search. Code for reproducing our experiments is available at https://github.com/BayesWatch/nas-without-training.

Accepted at ICML 2021 for a long presentation

References in corpus (10)

• Neural Architecture Search with Reinforcement Learning
• A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets
• NAS-Bench-101: Towards Reproducible Neural Architecture Search
• A Survey on Neural Architecture Search
• NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size
• Peephole: Predicting Network Performance Before Training
• NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search
• Pruning neural networks without any data by iteratively conserving synaptic flow
• Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective
• Towards NNGP-guided Neural Architecture Search

Cited by in corpus (21)

• Sustainable AI: Environmental Implications, Challenges and Opportunities
• Zero-Cost Proxies for Lightweight NAS
• A Comprehensive Survey on Hardware-Aware Neural Architecture Search
• Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective
• EPE-NAS: Efficient Performance Estimation Without Training for Neural Architecture Search
• StressNAS: Affect State and Stress Detection Using Neural Architecture Search
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• How Powerful are Performance Predictors in Neural Architecture Search?
• Towards NNGP-guided Neural Architecture Search
• Speedy Performance Estimation for Neural Architecture Search
• NASI: Label- and Data-agnostic Neural Architecture Search at Initialization
• Understanding and Accelerating Neural Architecture Search with Training-Free and Theory-Grounded Metrics
• AgEBO-Tabular: Joint Neural Architecture and Hyperparameter Search with Autotuned Data-Parallel Training for Tabular Data
• Privacy-preserving Collaborative Learning with Automatic Transformation Search
• Trainless Model Performance Estimation for Neural Architecture Search
• Generic Neural Architecture Search via Regression
• On the Orthogonality of Knowledge Distillation with Other Techniques: From an Ensemble Perspective
• Contrastive Embeddings for Neural Architectures
• Neural Architecture Search with Random Labels
• BaLeNAS: Differentiable Architecture Search via the Bayesian Learning Rule
• A Survey on Green Deep Learning