DARTS+: Improved Differentiable Architecture Search with Early Stopping

Abstract

Recently, there has been a growing interest in automating the process of neural architecture design, and the Differentiable Architecture Search (DARTS) method makes the process available within a few GPU days. However, the performance of DARTS is often observed to collapse when the number of search epochs becomes large. Meanwhile, lots of "{\em skip-connect}s" are found in the selected architectures. In this paper, we claim that the cause of the collapse is that there exists overfitting in the optimization of DARTS. Therefore, we propose a simple and effective algorithm, named "DARTS+", to avoid the collapse and improve the original DARTS, by "early stopping" the search procedure when meeting a certain criterion. We also conduct comprehensive experiments on benchmark datasets and different search spaces and show the effectiveness of our DARTS+ algorithm, and DARTS+ achieves $2.32\%$ test error on CIFAR10, $14.87\%$ on CIFAR100, and $23.7\%$ on ImageNet. We further remark that the idea of "early stopping" is implicitly included in some existing DARTS variants by manually setting a small number of search epochs, while we give an {\em explicit} criterion for "early stopping".

References in corpus (21)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Neural Architecture Search with Reinforcement Learning
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• DARTS: Differentiable Architecture Search
• AutoAugment: Learning Augmentation Policies from Data
• Efficient Neural Architecture Search via Parameter Sharing
• Neural Architecture Optimization
• PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
• SMASH: One-Shot Model Architecture Search through HyperNetworks
• Towards Principled Methods for Training Generative Adversarial Networks
• Understanding and Robustifying Differentiable Architecture Search
• Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation
• NAS evaluation is frustratingly hard
• Exploring Randomly Wired Neural Networks for Image Recognition
• ASAP: Architecture Search, Anneal and Prune
• The Early Phase of Neural Network Training
• AutoFIS: Automatic Feature Interaction Selection in Factorization Models for Click-Through Rate Prediction
• Theory-Inspired Path-Regularized Differential Network Architecture Search
• XNAS: Neural Architecture Search with Expert Advice
• Efficient Neural Architecture Search via Proximal Iterations
• Hierarchical Neural Architecture Search via Operator Clustering

Cited by in corpus (71)

• AutoML: A Survey of the State-of-the-Art
• Differentiable Quantum Architecture Search
• Review: Deep Learning in Electron Microscopy
• Stabilizing Differentiable Architecture Search via Perturbation-based Regularization
• ASAP: Architecture Search, Anneal and Prune
• ResKD: Residual-Guided Knowledge Distillation
• Retinex-inspired Unrolling with Cooperative Prior Architecture Search for Low-light Image Enhancement
• Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective
• Stabilizing DARTS with Amended Gradient Estimation on Architectural Parameters
• MANAS: Multi-Scale and Multi-Level Neural Architecture Search for Low-Dose CT Denoising
• Noisy Differentiable Architecture Search
• Blending Diverse Physical Priors with Neural Networks
• Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search
• DARTS-: Robustly Stepping out of Performance Collapse Without Indicators
• GOLD-NAS: Gradual, One-Level, Differentiable
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• Learning Invariances in Neural Networks
• Theory-Inspired Path-Regularized Differential Network Architecture Search
• How Powerful are Performance Predictors in Neural Architecture Search?
• Interpretable Neural Architecture Search via Bayesian Optimisation with Weisfeiler-Lehman Kernels
• A General Method For Automatic Discovery of Powerful Interactions In Click-Through Rate Prediction
• HardCoRe-NAS: Hard Constrained diffeRentiable Neural Architecture Search
• Efficient Differentiable Neural Architecture Search with Meta Kernels
• Rethinking Channel Dimensions for Efficient Model Design
• BETANAS: BalancEd TrAining and selective drop for Neural Architecture Search
• Less is More: Proxy Datasets in NAS approaches
• Cyclic Differentiable Architecture Search
• NASGEM: Neural Architecture Search via Graph Embedding Method
• Differentiable architecture search with multi-dimensional attention for spiking neural networks
• DSRNA: Differentiable Search of Robust Neural Architectures
• Hierarchical Neural Architecture Search via Operator Clustering
• TF-NAS: Rethinking Three Search Freedoms of Latency-Constrained Differentiable Neural Architecture Search
• Learning Versatile Neural Architectures by Propagating Network Codes
• Understanding and Accelerating Neural Architecture Search with Training-Free and Theory-Grounded Metrics
• Rethinking Graph Neural Architecture Search from Message-passing
• Darts-Conformer: Towards Efficient Gradient-Based Neural Architecture Search For End-to-End ASR
• NAS-Count: Counting-by-Density with Neural Architecture Search
• Geometry-Aware Gradient Algorithms for Neural Architecture Search
• An Asymptotically Optimal Multi-Armed Bandit Algorithm and Hyperparameter Optimization
• i-Razor: A Differentiable Neural Input Razor for Feature Selection and Dimension Search in DNN-Based Recommender Systems
• DOTS: Decoupling Operation and Topology in Differentiable Architecture Search
• A Value-Function-based Interior-point Method for Non-convex Bi-level Optimization
• RA-BNN: Constructing Robust & Accurate Binary Neural Network to Simultaneously Defend Adversarial Bit-Flip Attack and Improve Accuracy
• HR-NAS: Searching Efficient High-Resolution Neural Architectures with Lightweight Transformers
• Neighborhood-Aware Neural Architecture Search
• DA-NAS: Data Adapted Pruning for Efficient Neural Architecture Search
• Interleaving Learning, with Application to Neural Architecture Search
• Single-level Optimization For Differential Architecture Search
• Robustifying DARTS by Eliminating Information Bypass Leakage via Explicit Sparse Regularization
• Memory-Efficient Hierarchical Neural Architecture Search for Image Denoising
• Value-Function-based Sequential Minimization for Bi-level Optimization
• Learning by Self-Explanation, with Application to Neural Architecture Search
• BaLeNAS: Differentiable Architecture Search via the Bayesian Learning Rule
• MS-DARTS: Mean-Shift Based Differentiable Architecture Search
• iDARTS: Improving DARTS by Node Normalization and Decorrelation Discretization
• Disentangled Neural Architecture Search
• Neural Architecture Search as Sparse Supernet
• Single-DARTS: Towards Stable Architecture Search
• Edge-featured Graph Neural Architecture Search
• Neural Architecture Search in operational context: a remote sensing case-study
• Small-Group Learning, with Application to Neural Architecture Search
• Unchain the Search Space with Hierarchical Differentiable Architecture Search
• Searching Efficient Model-guided Deep Network for Image Denoising
• Understanding the wiring evolution in differentiable neural architecture search
• MSR-DARTS: Minimum Stable Rank of Differentiable Architecture Search
• Connection Sensitivity Matters for Training-free DARTS: From Architecture-Level Scoring to Operation-Level Sensitivity Analysis
• BNAS-v2: Memory-efficient and Performance-collapse-prevented Broad Neural Architecture Search
• Training BatchNorm Only in Neural Architecture Search and Beyond
• Learning from Mistakes based on Class Weighting with Application to Neural Architecture Search
• DARTS-PRIME: Regularization and Scheduling Improve Constrained Optimization in Differentiable NAS
• Mutually-aware Sub-Graphs Differentiable Architecture Search