Progressive DARTS: Bridging the Optimization Gap for NAS in the Wild

Abstract

With the rapid development of neural architecture search (NAS), researchers found powerful network architectures for a wide range of vision tasks. However, it remains unclear if the searched architecture can transfer across different types of tasks as manually designed ones did. This paper puts forward this problem, referred to as NAS in the wild, which explores the possibility of finding the optimal architecture in a proxy dataset and then deploying it to mostly unseen scenarios. We instantiate this setting using a currently popular algorithm named differentiable architecture search (DARTS), which often suffers unsatisfying performance while being transferred across different tasks. We argue that the accuracy drop originates from the formulation that uses a super-network for search but a sub-network for re-training. The different properties of these stages have resulted in a significant optimization gap, and consequently, the architectural parameters "over-fit" the super-network. To alleviate the gap, we present a progressive method that gradually increases the network depth during the search stage, which leads to the Progressive DARTS (P-DARTS) algorithm. With a reduced search cost (7 hours on a single GPU), P-DARTS achieves improved performance on both the proxy dataset (CIFAR10) and a few target problems (ImageNet classification, COCO detection and three ReID benchmarks). Our code is available at \url{https://github.com/chenxin061/pdarts}.

arXiv commentAn extension of P-DARTS. Previous version: arXiv:1904.12760

References in corpus (21)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• Improved Regularization of Convolutional Neural Networks with Cutout
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• MobileNetV2: Inverted Residuals and Linear Bottlenecks
• Wide Residual Networks
• Neural Architecture Search: A Survey
• DARTS: Differentiable Architecture Search
• Random Erasing Data Augmentation
• Efficient Neural Architecture Search via Parameter Sharing
• Efficient Architecture Search by Network Transformation
• ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware
• SNAS: Stochastic Neural Architecture Search
• ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design
• Unlabeled Samples Generated by GAN Improve the Person Re-identification Baseline in vitro
• CenterNet: Keypoint Triplets for Object Detection
• Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation
• A Genetic Programming Approach to Designing Convolutional Neural Network Architectures
• Stabilizing DARTS with Amended Gradient Estimation on Architectural Parameters

Cited by in corpus (8)

• CARS: Continuous Evolution for Efficient Neural Architecture Search
• Trilevel Neural Architecture Search for Efficient Single Image Super-Resolution
• FBNetV5: Neural Architecture Search for Multiple Tasks in One Run
• Zen-NAS: A Zero-Shot NAS for High-Performance Deep Image Recognition
• Few-shot Neural Architecture Search
• DSRNA: Differentiable Search of Robust Neural Architectures
• NAX: Co-Designing Neural Network and Hardware Architecture for Memristive Xbar based Computing Systems
• Parameter Prediction for Unseen Deep Architectures