Rethinking Architecture Selection in Differentiable NAS

Abstract

Differentiable Neural Architecture Search is one of the most popular Neural Architecture Search (NAS) methods for its search efficiency and simplicity, accomplished by jointly optimizing the model weight and architecture parameters in a weight-sharing supernet via gradient-based algorithms. At the end of the search phase, the operations with the largest architecture parameters will be selected to form the final architecture, with the implicit assumption that the values of architecture parameters reflect the operation strength. While much has been discussed about the supernet's optimization, the architecture selection process has received little attention. We provide empirical and theoretical analysis to show that the magnitude of architecture parameters does not necessarily indicate how much the operation contributes to the supernet's performance. We propose an alternative perturbation-based architecture selection that directly measures each operation's influence on the supernet. We re-evaluate several differentiable NAS methods with the proposed architecture selection and find that it is able to extract significantly improved architectures from the underlying supernets consistently. Furthermore, we find that several failure modes of DARTS can be greatly alleviated with the proposed selection method, indicating that much of the poor generalization observed in DARTS can be attributed to the failure of magnitude-based architecture selection rather than entirely the optimization of its supernet.

Outstanding Paper Award at ICLR 2021. The code is available at https://github.com/ruocwang/darts-pt

References in corpus (7)

• DARTS: Differentiable Architecture Search
• Large-Scale Evolution of Image Classifiers
• Neural Architecture Optimization
• PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
• Highway and Residual Networks learn Unrolled Iterative Estimation
• BayesNAS: A Bayesian Approach for Neural Architecture Search
• Stabilizing DARTS with Amended Gradient Estimation on Architectural Parameters

Cited by in corpus (13)

• DASS: Differentiable Architecture Search for Sparse neural networks
• Learning Versatile Neural Architectures by Propagating Network Codes
• DrNAS: Dirichlet Neural Architecture Search
• On the Privacy Risks of Cell-Based NAS Architectures
• Bag of Tricks for Neural Architecture Search
• FEAR: A Simple Lightweight Method to Rank Architectures
• Learning Graph Representation of Person-specific Cognitive Processes from Audio-visual Behaviours for Automatic Personality Recognition
• BaLeNAS: Differentiable Architecture Search via the Bayesian Learning Rule
• Zero-Cost Operation Scoring in Differentiable Architecture Search
• iDARTS: Improving DARTS by Node Normalization and Decorrelation Discretization
• On the Security Risks of AutoML
• DARTS-PRIME: Regularization and Scheduling Improve Constrained Optimization in Differentiable NAS
• Connection Sensitivity Matters for Training-free DARTS: From Architecture-Level Scoring to Operation-Level Sensitivity Analysis