Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search
arXiv:1911.12126
Abstract
Differentiable Architecture Search (DARTS) is now a widely disseminated weight-sharing neural architecture search method. However, it suffers from well-known performance collapse due to an inevitable aggregation of skip connections. In this paper, we first disclose that its root cause lies in an unfair advantage in exclusive competition. Through experiments, we show that if either of two conditions is broken, the collapse disappears. Thereby, we present a novel approach called Fair DARTS where the exclusive competition is relaxed to be collaborative. Specifically, we let each operation's architectural weight be independent of others. Yet there is still an important issue of discretization discrepancy. We then propose a zero-one loss to push architectural weights towards zero or one, which approximates an expected multi-hot solution. Our experiments are performed on two mainstream search spaces, and we derive new state-of-the-art results on CIFAR-10 and ImageNet. Our code is available on https://github.com/xiaomi-automl/fairdarts .
arXiv commentAccepted to ECCV 2020, camera ready version
References in corpus (10)
- Categorical Reparameterization with Gumbel-Softmax
- DARTS: Differentiable Architecture Search
- The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
- MMDetection: Open MMLab Detection Toolbox and Benchmark
- Searching for Activation Functions
- AutoAugment: Learning Augmentation Policies from Data
- MixConv: Mixed Depthwise Convolutional Kernels
- DARTS+: Improved Differentiable Architecture Search with Early Stopping
- One-Shot Neural Architecture Search via Self-Evaluated Template Network
- Stabilizing DARTS with Amended Gradient Estimation on Architectural Parameters