Understanding and Robustifying Differentiable Architecture Search

Abstract

Differentiable Architecture Search (DARTS) has attracted a lot of attention due to its simplicity and small search costs achieved by a continuous relaxation and an approximation of the resulting bi-level optimization problem. However, DARTS does not work robustly for new problems: we identify a wide range of search spaces for which DARTS yields degenerate architectures with very poor test performance. We study this failure mode and show that, while DARTS successfully minimizes validation loss, the found solutions generalize poorly when they coincide with high validation loss curvature in the architecture space. We show that by adding one of various types of regularization we can robustify DARTS to find solutions with less curvature and better generalization properties. Based on these observations, we propose several simple variations of DARTS that perform substantially more robustly in practice. Our observations are robust across five search spaces on three image classification tasks and also hold for the very different domains of disparity estimation (a dense regression task) and language modelling.

In: International Conference on Learning Representations (ICLR 2020); 28 pages, 30 figures

References in corpus (15)

• Neural Architecture Search with Reinforcement Learning
• Improved Regularization of Convolutional Neural Networks with Cutout
• Neural Architecture Search: A Survey
• DARTS: Differentiable Architecture Search
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Large-Scale Evolution of Image Classifiers
• Designing Neural Network Architectures using Reinforcement Learning
• Efficient Architecture Search by Network Transformation
• Random Search and Reproducibility for Neural Architecture Search
• Convolutional Neural Fabrics
• Path-Level Network Transformation for Efficient Architecture Search
• Hessian-based Analysis of Large Batch Training and Robustness to Adversaries
• Best Practices for Scientific Research on Neural Architecture Search
• Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search
• Probabilistic Neural Architecture Search

Cited by in corpus (67)

• AutoML: A Survey of the State-of-the-Art
• DARTS+: Improved Differentiable Architecture Search with Early Stopping
• A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions
• Differentiable Quantum Architecture Search
• When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations
• Best Practices for Scientific Research on Neural Architecture Search
• Stabilizing Differentiable Architecture Search via Perturbation-based Regularization
• NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search
• EPE-NAS: Efficient Performance Estimation Without Training for Neural Architecture Search
• DAIS: Automatic Channel Pruning via Differentiable Annealing Indicator Search
• Stabilizing DARTS with Amended Gradient Estimation on Architectural Parameters
• Noisy Differentiable Architecture Search
• Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?
• DARTS-: Robustly Stepping out of Performance Collapse Without Indicators
• Theory-Inspired Path-Regularized Differential Network Architecture Search
• AutoHAS: Efficient Hyperparameter and Architecture Search
• How to Train Your Super-Net: An Analysis of Training Heuristics in Weight-Sharing NAS
• Differential Evolution for Neural Architecture Search
• Neural Ensemble Search for Uncertainty Estimation and Dataset Shift
• How Powerful are Performance Predictors in Neural Architecture Search?
• MixPath: A Unified Approach for One-shot Neural Architecture Search
• Interstellar: Searching Recurrent Architecture for Knowledge Graph Embedding
• DASS: Differentiable Architecture Search for Sparse neural networks
• Less is More: Proxy Datasets in NAS approaches
• Local Learning Matters: Rethinking Data Heterogeneity in Federated Learning
• DSRNA: Differentiable Search of Robust Neural Architectures
• AlphaGAN: Fully Differentiable Architecture Search for Generative Adversarial Networks
• Auto-PyTorch Tabular: Multi-Fidelity MetaLearning for Efficient and Robust AutoDL
• Hierarchical Neural Architecture Search via Operator Clustering
• Few-shot Neural Architecture Search
• CATE: Computation-aware Neural Architecture Encoding with Transformers
• NASI: Label- and Data-agnostic Neural Architecture Search at Initialization
• Rethinking Graph Neural Architecture Search from Message-passing
• DrNAS: Dirichlet Neural Architecture Search
• Accuracy Prediction with Non-neural Model for Neural Architecture Search
• HourNAS: Extremely Fast Neural Architecture Search Through an Hourglass Lens
• VINNAS: Variational Inference-based Neural Network Architecture Search
• An Asymptotically Optimal Multi-Armed Bandit Algorithm and Hyperparameter Optimization
• DOTS: Decoupling Operation and Topology in Differentiable Architecture Search
• ZARTS: On Zero-order Optimization for Neural Architecture Search
• Geometry-Aware Gradient Algorithms for Neural Architecture Search
• Auto-Classifier: A Robust Defect Detector Based on an AutoML Head
• DAAS: Differentiable Architecture and Augmentation Policy Search
• SCARLET-NAS: Bridging the Gap between Stability and Scalability in Weight-sharing Neural Architecture Search
• Effective, Efficient and Robust Neural Architecture Search
• Operation-level Progressive Differentiable Architecture Search
• Generic Neural Architecture Search via Regression
• Neighborhood-Aware Neural Architecture Search
• Single-level Optimization For Differential Architecture Search
• Robustifying DARTS by Eliminating Information Bypass Leakage via Explicit Sparse Regularization
• Inception Convolution with Efficient Dilation Search
• Hyperparameter optimization with REINFORCE and Transformers
• ProxyBO: Accelerating Neural Architecture Search via Bayesian Optimization with Zero-cost Proxies
• BaLeNAS: Differentiable Architecture Search via the Bayesian Learning Rule
• MS-DARTS: Mean-Shift Based Differentiable Architecture Search
• Landmark Regularization: Ranking Guided Super-Net Training in Neural Architecture Search
• Zero-Cost Operation Scoring in Differentiable Architecture Search
• Gradient-based Hyperparameter Optimization Over Long Horizons
• Discretization-Aware Architecture Search
• Global2Local: Efficient Structure Search for Video Action Segmentation
• Connection Sensitivity Matters for Training-free DARTS: From Architecture-Level Scoring to Operation-Level Sensitivity Analysis
• A Survey on Green Deep Learning
• Improving Differentiable Architecture Search with a Generative Model
• Learning from Mistakes based on Class Weighting with Application to Neural Architecture Search
• Mutually-aware Sub-Graphs Differentiable Architecture Search
• Edge-featured Graph Neural Architecture Search
• Making Differentiable Architecture Search less local