Efficient Neural Architecture Search via Proximal Iterations
arXiv:1905.13577
Abstract
Neural architecture search (NAS) recently attracts much research attention because of its ability to identify better architectures than handcrafted ones. However, many NAS methods, which optimize the search process in a discrete search space, need many GPU days for convergence. Recently, DARTS, which constructs a differentiable search space and then optimizes it by gradient descent, can obtain high-performance architecture and reduces the search time to several days. However, DARTS is still slow as it updates an ensemble of all operations and keeps only one after convergence. Besides, DARTS can converge to inferior architectures due to the strong correlation among operations. In this paper, we propose a new differentiable Neural Architecture Search method based on Proximal gradient descent (denoted as NASP). Different from DARTS, NASP reformulates the search process as an optimization problem with a constraint that only one operation is allowed to be updated during forward and backward propagation. Since the constraint is hard to deal with, we propose a new algorithm inspired by proximal iterations to solve it. Experiments on various tasks demonstrate that NASP can obtain high-performance architectures with 10 times of speedup on the computational time than DARTS.
arXiv commentAccepted by AAAI-2020
References in corpus (18)
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Neural Architecture Search with Reinforcement Learning
- Categorical Reparameterization with Gumbel-Softmax
- Improved Regularization of Convolutional Neural Networks with Cutout
- DARTS: Differentiable Architecture Search
- QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
- AutoAugment: Learning Augmentation Policies from Data
- Neural Architecture Optimization
- Learning Structured Sparsity in Deep Neural Networks
- Regularizing and Optimizing LSTM Language Models
- Designing Neural Network Architectures using Reinforcement Learning
- Predictive Uncertainty Estimation via Prior Networks
- Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling
- Loss-aware Binarization of Deep Networks
- BayesNAS: A Bayesian Approach for Neural Architecture Search
- ProxQuant: Quantized Neural Networks via Proximal Operators
- ASAP: Architecture Search, Anneal and Prune
- Adaptive Stochastic Natural Gradient Method for One-Shot Neural Architecture Search