One-Shot Neural Architecture Search via Compressive Sensing
arXiv:1906.02869
Abstract
Neural Architecture Search remains a very challenging meta-learning problem. Several recent techniques based on parameter-sharing idea have focused on reducing the NAS running time by leveraging proxy models, leading to architectures with competitive performance compared to those with hand-crafted designs. In this paper, we propose an iterative technique for NAS, inspired by algorithms for learning low-degree sparse Boolean functions. We validate our approach on the DARTs search space (Liu et al., 2018b) and NAS-Bench-201 (Yang et al., 2020). In addition, we provide theoretical analysis via upper bounds on the number of validation error measurements needed for reliable learning, and include ablation studies to further in-depth understanding of our technique.
2nd Workshop on Neural Architecture Search at ICLR 2021
References in corpus (9)
- Neural Architecture Search with Reinforcement Learning
- Improved Regularization of Convolutional Neural Networks with Cutout
- Efficient Architecture Search by Network Transformation
- SNAS: Stochastic Neural Architecture Search
- Random Search and Reproducibility for Neural Architecture Search
- Evaluating the Search Phase of Neural Architecture Search
- Exploring Randomly Wired Neural Networks for Image Recognition
- Hyperparameter Optimization: A Spectral Approach
- Learnable Embedding Space for Efficient Neural Architecture Compression
Cited by in corpus (7)
- ISTA-NAS: Efficient and Consistent Neural Architecture Search by Sparse Coding
- Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
- Sphynx: ReLU-Efficient Network Design for Private Inference
- Single-Path Mobile AutoML: Efficient ConvNet Design and NAS Hyperparameter Optimization
- Fisher Task Distance and Its Application in Neural Architecture Search
- Neural Architecture Search as Sparse Supernet
- Hyperparameter Optimization in Neural Networks via Structured Sparse Recovery