Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation

Abstract

Recently, differentiable search methods have made major progress in reducing the computational costs of neural architecture search. However, these approaches often report lower accuracy in evaluating the searched architecture or transferring it to another dataset. This is arguably due to the large gap between the architecture depths in search and evaluation scenarios. In this paper, we present an efficient algorithm which allows the depth of searched architectures to grow gradually during the training procedure. This brings two issues, namely, heavier computational overheads and weaker search stability, which we solve using search space approximation and regularization, respectively. With a significantly reduced search time (~7 hours on a single GPU), our approach achieves state-of-the-art performance on both the proxy dataset (CIFAR10 or CIFAR100) and the target dataset (ImageNet). Code is available at https://github.com/chenxin061/pdarts.

10 pages, 3 figures, 3 tables

References in corpus (4)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Improved Regularization of Convolutional Neural Networks with Cutout
• Designing Neural Network Architectures using Reinforcement Learning
• Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation

Cited by in corpus (15)

• XNAS: Neural Architecture Search with Expert Advice
• Progressive DARTS: Bridging the Optimization Gap for NAS in the Wild
• Efficient Differentiable Neural Architecture Search with Meta Kernels
• Fine-Grained Neural Architecture Search
• BETANAS: BalancEd TrAining and selective drop for Neural Architecture Search
• Multi-objective Neural Architecture Search via Non-stationary Policy Gradient
• ModuleNet: Knowledge-inherited Neural Architecture Search
• EDAS: Efficient and Differentiable Architecture Search
• Exploiting Operation Importance for Differentiable Neural Architecture Search
• RAPDARTS: Resource-Aware Progressive Differentiable Architecture Search
• Binarized Neural Architecture Search
• Disentangled Neural Architecture Search
• ADWPNAS: Architecture-Driven Weight Prediction for Neural Architecture Search
• Adaptive Interaction Modeling via Graph Operations Search
• Network Adjustment: Channel Search Guided by FLOPs Utilization Ratio