DARTS-: Robustly Stepping out of Performance Collapse Without Indicators

Abstract

Despite the fast development of differentiable architecture search (DARTS), it suffers from long-standing performance instability, which extremely limits its application. Existing robustifying methods draw clues from the resulting deteriorated behavior instead of finding out its causing factor. Various indicators such as Hessian eigenvalues are proposed as a signal to stop searching before the performance collapses. However, these indicator-based methods tend to easily reject good architectures if the thresholds are inappropriately set, let alone the searching is intrinsically noisy. In this paper, we undertake a more subtle and direct approach to resolve the collapse. We first demonstrate that skip connections have a clear advantage over other candidate operations, where it can easily recover from a disadvantageous state and become dominant. We conjecture that this privilege is causing degenerated performance. Therefore, we propose to factor out this benefit with an auxiliary skip connection, ensuring a fairer competition for all operations. We call this approach DARTS-. Extensive experiments on various datasets verify that it can substantially improve robustness. Our code is available at https://github.com/Meituan-AutoML/DARTS- .

arXiv commentAccepted to ICLR2021

References in corpus (15)

• DARTS: Differentiable Architecture Search
• MMDetection: Open MMLab Detection Toolbox and Benchmark
• AutoAugment: Learning Augmentation Policies from Data
• Efficient Neural Architecture Search via Parameter Sharing
• Boosting the accuracy of multi-spectral image pan-sharpening by learning a deep residual network
• PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
• MixConv: Mixed Depthwise Convolutional Kernels
• Understanding and Robustifying Differentiable Architecture Search
• DARTS+: Improved Differentiable Architecture Search with Early Stopping
• One-Shot Neural Architecture Search via Self-Evaluated Template Network
• The Evolved Transformer
• NAS evaluation is frustratingly hard
• Stabilizing DARTS with Amended Gradient Estimation on Architectural Parameters
• AtomNAS: Fine-Grained End-to-End Neural Architecture Search
• Theory-Inspired Path-Regularized Differential Network Architecture Search

Cited by in corpus (5)

• NASI: Label- and Data-agnostic Neural Architecture Search at Initialization
• DOTS: Decoupling Operation and Topology in Differentiable Architecture Search
• Single-level Optimization For Differential Architecture Search
• Edge-featured Graph Neural Architecture Search
• Mutually-aware Sub-Graphs Differentiable Architecture Search