Noisy Differentiable Architecture Search

Abstract

Simplicity is the ultimate sophistication. Differentiable Architecture Search (DARTS) has now become one of the mainstream paradigms of neural architecture search. However, it largely suffers from the well-known performance collapse issue due to the aggregation of skip connections. It is thought to have overly benefited from the residual structure which accelerates the information flow. To weaken this impact, we propose to inject unbiased random noise to impede the flow. We name this novel approach NoisyDARTS. In effect, a network optimizer should perceive this difficulty at each training step and refrain from overshooting, especially on skip connections. In the long run, since we add no bias to the gradient in terms of expectation, it is still likely to converge to the right solution area. We also prove that the injected noise plays a role in smoothing the loss landscape, which makes the optimization easier. Our method features extreme simplicity and acts as a new strong baseline. We perform extensive experiments across various search spaces, datasets, and tasks, where we robustly achieve state-of-the-art results. Our code is available at https://github.com/xiaomi-automl/NoisyDARTS.

BMVC 2021

References in corpus (6)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Improving neural networks by preventing co-adaptation of feature detectors
• MMDetection: Open MMLab Detection Toolbox and Benchmark
• Adversarial Weight Perturbation Helps Robust Generalization
• Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation
• Theory-Inspired Path-Regularized Differential Network Architecture Search

Cited by in corpus (9)

• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• A General Method For Automatic Discovery of Powerful Interactions In Click-Through Rate Prediction
• VINNAS: Variational Inference-based Neural Network Architecture Search
• AutoKWS: Keyword Spotting with Differentiable Architecture Search
• Encoder-Decoder Neural Architecture Optimization for Keyword Spotting
• Deep Neural Networks with Short Circuits for Improved Gradient Learning
• A Survey on Green Deep Learning
• Unchain the Search Space with Hierarchical Differentiable Architecture Search
• Reconsidering CO2 emissions from Computer Vision