Stabilizing Differentiable Architecture Search via Perturbation-based Regularization

Abstract

Differentiable architecture search (DARTS) is a prevailing NAS solution to identify architectures. Based on the continuous relaxation of the architecture space, DARTS learns a differentiable architecture weight and largely reduces the search cost. However, its stability has been challenged for yielding deteriorating architectures as the search proceeds. We find that the precipitous validation loss landscape, which leads to a dramatic performance drop when distilling the final architecture, is an essential factor that causes instability. Based on this observation, we propose a perturbation-based regularization - SmoothDARTS (SDARTS), to smooth the loss landscape and improve the generalizability of DARTS-based methods. In particular, our new formulations stabilize DARTS-based methods by either random smoothing or adversarial attack. The search trajectory on NAS-Bench-1Shot1 demonstrates the effectiveness of our approach and due to the improved stability, we achieve performance gain across various search spaces on 4 datasets. Furthermore, we mathematically show that SDARTS implicitly regularizes the Hessian norm of the validation loss, which accounts for a smoother loss landscape and improved performance.

ICML 2020, code is available at https://github.com/xiangning-chen/SmoothDARTS

References in corpus (19)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Neural Architecture Search with Reinforcement Learning
• Improved Regularization of Convolutional Neural Networks with Cutout
• DARTS: Differentiable Architecture Search
• Certified Adversarial Robustness via Randomized Smoothing
• Large-Scale Evolution of Image Classifiers
• Neural Architecture Optimization
• PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
• Efficient Architecture Search by Network Transformation
• ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware
• SNAS: Stochastic Neural Architecture Search
• Understanding and Robustifying Differentiable Architecture Search
• DARTS+: Improved Differentiable Architecture Search with Early Stopping
• Random Search and Reproducibility for Neural Architecture Search
• Certified Adversarial Robustness with Additive Noise
• Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation
• BayesNAS: A Bayesian Approach for Neural Architecture Search
• NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search
• Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search

Cited by in corpus (10)

• Differentiable Quantum Architecture Search
• Partial Connection Based on Channel Attention for Differentiable Neural Architecture Search
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• DSRNA: Differentiable Search of Robust Neural Architectures
• Understanding and Accelerating Neural Architecture Search with Training-Free and Theory-Grounded Metrics
• On the Privacy Risks of Cell-Based NAS Architectures
• BARS: Joint Search of Cell Topology and Layout for Accurate and Efficient Binary ARchitectures
• Operation-level Progressive Differentiable Architecture Search
• Robustifying DARTS by Eliminating Information Bypass Leakage via Explicit Sparse Regularization
• Mutually-aware Sub-Graphs Differentiable Architecture Search