Prune and Replace NAS
arXiv:1906.07528 · doi:10.1109/ICMLA.2019.00158
Abstract
While recent NAS algorithms are thousands of times faster than the pioneering works, it is often overlooked that they use fewer candidate operations, resulting in a significantly smaller search space. We present PR-DARTS, a NAS algorithm that discovers strong network configurations in a much larger search space and a single day. A small candidate operation pool is used, from which candidates are progressively pruned and replaced with better performing ones. Experiments on CIFAR-10 and CIFAR-100 achieve 2.51% and 15.53% test error, respectively, despite searching in a space where each cell has 150 times as many possible configurations than in the DARTS baseline. Code is available at https://github.com/cogsys-tuebingen/prdarts
9 pages, 3 figures, 3 tables reworked, accepted at the ICMLA 2019
References in corpus (13)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- mixup: Beyond Empirical Risk Minimization
- Neural Architecture Search with Reinforcement Learning
- Improved Regularization of Convolutional Neural Networks with Cutout
- DARTS: Differentiable Architecture Search
- The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
- SNAS: Stochastic Neural Architecture Search
- NAS-Bench-101: Towards Reproducible Neural Architecture Search
- Simple And Efficient Architecture Search for Convolutional Neural Networks
- Path-Level Network Transformation for Efficient Architecture Search
- Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation
- ASAP: Architecture Search, Anneal and Prune
- sharpDARTS: Faster and More Accurate Differentiable Architecture Search