MiLeNAS: Efficient Neural Architecture Search via Mixed-Level Reformulation
arXiv:2003.12238
Abstract
Many recently proposed methods for Neural Architecture Search (NAS) can be formulated as bilevel optimization. For efficient implementation, its solution requires approximations of second-order methods. In this paper, we demonstrate that gradient errors caused by such approximations lead to suboptimality, in the sense that the optimization procedure fails to converge to a (locally) optimal solution. To remedy this, this paper proposes \mldas, a mixed-level reformulation for NAS that can be optimized efficiently and reliably. It is shown that even when using a simple first-order method on the mixed-level formulation, \mldas\ can achieve a lower validation error for NAS problems. Consequently, architectures obtained by our method achieve consistently higher accuracies than those obtained from bilevel optimization. Moreover, \mldas\ proposes a framework beyond DARTS. It is upgraded via model size-based search and early stopping strategies to complete the search process in around 5 hours. Extensive experiments within the convolutional architecture search space validate the effectiveness of our approach.
arXiv commentThis paper is published in CVPR 2020 (IEEE/CVF Conference on Computer Vision and Pattern Recognition 2020)
References in corpus (14)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- DeepPose: Human Pose Estimation via Deep Neural Networks
- Large-Scale Evolution of Image Classifiers
- Neural Architecture Optimization
- SNAS: Stochastic Neural Architecture Search
- Hierarchical Representations for Efficient Architecture Search
- Neural Optimizer Search with Reinforcement Learning
- Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation
- sharpDARTS: Faster and More Accurate Differentiable Architecture Search
- CARS: Continuous Evolution for Efficient Neural Architecture Search
- XNAS: Neural Architecture Search with Expert Advice
- BlockQNN: Efficient Block-wise Neural Network Architecture Generation