Stabilizing DARTS with Amended Gradient Estimation on Architectural Parameters

Abstract

DARTS is a popular algorithm for neural architecture search (NAS). Despite its great advantage in search efficiency, DARTS often suffers weak stability, which reflects in the large variation among individual trials as well as the sensitivity to the hyper-parameters of the search process. This paper owes such instability to an optimization gap between the super-network and its sub-networks, namely, improving the validation accuracy of the super-network does not necessarily lead to a higher expectation on the performance of the sampled sub-networks. Then, we point out that the gap is due to the inaccurate estimation of the architectural gradients, based on which we propose an amended estimation method. Mathematically, our method guarantees a bounded error from the true gradients while the original estimation does not. Our approach bridges the gap from two aspects, namely, amending the estimation on the architectural gradients, and unifying the hyper-parameter settings in the search and re-training stages. Experiments on CIFAR10 and ImageNet demonstrate that our approach largely improves search stability and, more importantly, enables DARTS-based approaches to explore much larger search spaces that have not been investigated before.

22 pages, 12 figures, submitted to ICML 2020, updated experiments on Penn Treebank

References in corpus (19)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Neural Architecture Search with Reinforcement Learning
• Improved Regularization of Convolutional Neural Networks with Cutout
• DARTS: Differentiable Architecture Search
• AutoAugment: Learning Augmentation Policies from Data
• Large-Scale Evolution of Image Classifiers
• Neural Architecture Optimization
• PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
• Efficient Architecture Search by Network Transformation
• ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware
• SNAS: Stochastic Neural Architecture Search
• Understanding and Robustifying Differentiable Architecture Search
• DARTS+: Improved Differentiable Architecture Search with Early Stopping
• Random Search and Reproducibility for Neural Architecture Search
• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• Evaluating the Search Phase of Neural Architecture Search
• Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation
• ASAP: Architecture Search, Anneal and Prune
• XNAS: Neural Architecture Search with Expert Advice

Cited by in corpus (12)

• Rethinking Architecture Selection in Differentiable NAS
• Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search
• DARTS-: Robustly Stepping out of Performance Collapse Without Indicators
• GOLD-NAS: Gradual, One-Level, Differentiable
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• Progressive DARTS: Bridging the Optimization Gap for NAS in the Wild
• Less is More: Proxy Datasets in NAS approaches
• Joint Search of Data Augmentation Policies and Network Architectures
• ZARTS: On Zero-order Optimization for Neural Architecture Search
• Auto-Agent-Distiller: Towards Efficient Deep Reinforcement Learning Agents via Neural Architecture Search
• Direct Differentiable Augmentation Search
• iDARTS: Improving DARTS by Node Normalization and Decorrelation Discretization