RAPDARTS: Resource-Aware Progressive Differentiable Architecture Search
arXiv:1911.05704
Abstract
Early neural network architectures were designed by so-called "grad student descent". Since then, the field of Neural Architecture Search (NAS) has developed with the goal of algorithmically designing architectures tailored for a dataset of interest. Recently, gradient-based NAS approaches have been created to rapidly perform the search. Gradient-based approaches impose more structure on the search, compared to alternative NAS methods, enabling faster search phase optimization. In the real-world, neural architecture performance is measured by more than just high accuracy. There is increasing need for efficient neural architectures, where resources such as model size or latency must also be considered. Gradient-based NAS is also suitable for such multi-objective optimization. In this work we extend a popular gradient-based NAS method to support one or more resource costs. We then perform in-depth analysis on the discovery of architectures satisfying single-resource constraints for classification of CIFAR-10.
References in corpus (9)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Neural Architecture Search with Reinforcement Learning
- Large-Scale Evolution of Image Classifiers
- SNAS: Stochastic Neural Architecture Search
- Random Search and Reproducibility for Neural Architecture Search
- Single Path One-Shot Neural Architecture Search with Uniform Sampling
- A Survey on Neural Architecture Search
- Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation
- You Only Search Once: Single Shot Neural Architecture Search via Direct Sparse Optimization