Geometry-Aware Gradient Algorithms for Neural Architecture Search
arXiv:2004.07802
Abstract
Recent state-of-the-art methods for neural architecture search (NAS) exploit gradient-based optimization by relaxing the problem into continuous optimization over architectures and shared-weights, a noisy process that remains poorly understood. We argue for the study of single-level empirical risk minimization to understand NAS with weight-sharing, reducing the design of NAS methods to devising optimizers and regularizers that can quickly obtain high-quality solutions to this problem. Invoking the theory of mirror descent, we present a geometry-aware framework that exploits the underlying structure of this optimization to return sparse architectural parameters, leading to simple yet novel algorithms that enjoy fast convergence guarantees and achieve state-of-the-art accuracy on the latest NAS benchmarks in computer vision. Notably, we exceed the best published results for both CIFAR and ImageNet on both the DARTS search space and NAS-Bench201; on the latter we achieve near-oracle-optimal performance on CIFAR-10 and CIFAR-100. Together, our theory and experiments demonstrate a principled way to co-design optimizers and continuous relaxations of discrete NAS search spaces.
arXiv commentICLR 2021 Camera-Ready
References in corpus (14)
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Improved Regularization of Convolutional Neural Networks with Cutout
- DARTS: Differentiable Architecture Search
- Searching for Activation Functions
- PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
- Understanding and Robustifying Differentiable Architecture Search
- DARTS+: Improved Differentiable Architecture Search with Early Stopping
- ASAP: Architecture Search, Anneal and Prune
- Adaptive Stochastic Natural Gradient Method for One-Shot Neural Architecture Search
- On the Convergence Rate of Stochastic Mirror Descent for Nonsmooth Nonconvex Optimization
- sharpDARTS: Faster and More Accurate Differentiable Architecture Search
- Deeper Insights into Weight Sharing in Neural Architecture Search
- To Share or Not To Share: A Comprehensive Appraisal of Weight-Sharing
- Prune and Replace NAS