Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation

Abstract

Recently, differentiable search methods have made major progress in reducing the computational costs of neural architecture search. However, these approaches often report lower accuracy in evaluating the searched architecture or transferring it to another dataset. This is arguably due to the large gap between the architecture depths in search and evaluation scenarios. In this paper, we present an efficient algorithm which allows the depth of searched architectures to grow gradually during the training procedure. This brings two issues, namely, heavier computational overheads and weaker search stability, which we solve using search space approximation and regularization, respectively. With a significantly reduced search time (~7 hours on a single GPU), our approach achieves state-of-the-art performance on both the proxy dataset (CIFAR10 or CIFAR100) and the target dataset (ImageNet). Code is available at https://github.com/chenxin061/pdarts.

10 pages, 3 figures, 3 tables

References in corpus (10)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Improved Regularization of Convolutional Neural Networks with Cutout
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Neural Architecture Search: A Survey
• Training Very Deep Networks
• Efficient Neural Architecture Search via Parameter Sharing
• Designing Neural Network Architectures using Reinforcement Learning
• SNAS: Stochastic Neural Architecture Search
• Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation

Cited by in corpus (60)

• PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
• Stabilizing Differentiable Architecture Search via Perturbation-based Regularization
• BNAS:An Efficient Neural Architecture Search Approach Using Broad Scalable Architecture
• ASAP: Architecture Search, Anneal and Prune
• Semi-Supervised Neural Architecture Search
• Neural Architecture Search For Keyword Spotting
• Noisy Differentiable Architecture Search
• Latency-Aware Differentiable Neural Architecture Search
• Theory-Inspired Path-Regularized Differential Network Architecture Search
• XNAS: Neural Architecture Search with Expert Advice
• Progressive DARTS: Bridging the Optimization Gap for NAS in the Wild
• Machine Learning for the Detection and Identification of Internet of Things (IoT) Devices: A Survey
• Angle-based Search Space Shrinking for Neural Architecture Search
• Hit-Detector: Hierarchical Trinity Architecture Search for Object Detection
• Efficient Differentiable Neural Architecture Search with Meta Kernels
• AdversarialNAS: Adversarial Neural Architecture Search for GANs
• Fine-Grained Neural Architecture Search
• BETANAS: BalancEd TrAining and selective drop for Neural Architecture Search
• SGAS: Sequential Greedy Architecture Search
• Cyclic Differentiable Architecture Search
• Any-Precision Deep Neural Networks
• Balanced One-shot Neural Architecture Optimization
• MiLeNAS: Efficient Neural Architecture Search via Mixed-Level Reformulation
• Prune and Replace NAS
• DHA: End-to-End Joint Optimization of Data Augmentation Policy, Hyper-parameter and Architecture
• Joint Search of Data Augmentation Policies and Network Architectures
• Self-supervised Representation Learning for Evolutionary Neural Architecture Search
• Graph-guided Architecture Search for Real-time Semantic Segmentation
• ModuleNet: Knowledge-inherited Neural Architecture Search
• Multi-objective Neural Architecture Search via Non-stationary Policy Gradient
• Accuracy Prediction with Non-neural Model for Neural Architecture Search
• NAS-Count: Counting-by-Density with Neural Architecture Search
• ZARTS: On Zero-order Optimization for Neural Architecture Search
• Neural Architecture Refinement: A Practical Way for Avoiding Overfitting in NAS
• Gradient-based algorithms for multi-objective bi-level optimization
• EDAS: Efficient and Differentiable Architecture Search
• A Value-Function-based Interior-point Method for Non-convex Bi-level Optimization
• DAAS: Differentiable Architecture and Augmentation Policy Search
• Exploiting Operation Importance for Differentiable Neural Architecture Search
• RAPDARTS: Resource-Aware Progressive Differentiable Architecture Search
• Binarized Neural Architecture Search
• Neighborhood-Aware Neural Architecture Search
• SCARLET-NAS: Bridging the Gap between Stability and Scalability in Weight-sharing Neural Architecture Search
• Memory-Efficient Differentiable Transformer Architecture Search
• Single-level Optimization For Differential Architecture Search
• Neural Architecture Search as Sparse Supernet
• BN-NAS: Neural Architecture Search with Batch Normalization
• iDARTS: Improving DARTS by Node Normalization and Decorrelation Discretization
• Learning by Self-Explanation, with Application to Neural Architecture Search
• ADWPNAS: Architecture-Driven Weight Prediction for Neural Architecture Search
• Disentangled Neural Architecture Search
• Adaptive Interaction Modeling via Graph Operations Search
• Small-Group Learning, with Application to Neural Architecture Search
• One-Shot Neural Ensemble Architecture Search by Diversity-Guided Search Space Shrinking
• Unchain the Search Space with Hierarchical Differentiable Architecture Search
• Learning from Mistakes based on Class Weighting with Application to Neural Architecture Search
• Mutually-aware Sub-Graphs Differentiable Architecture Search
• Evolutionary NAS with Gene Expression Programming of Cellular Encoding
• Network Adjustment: Channel Search Guided by FLOPs Utilization Ratio
• Self-Supervised Neural Architecture Search for Imbalanced Datasets