Robustifying DARTS by Eliminating Information Bypass Leakage via Explicit Sparse Regularization
arXiv:2306.06858 · doi:10.1109/ICDM51629.2021.00099
Abstract
Differentiable architecture search (DARTS) is a promising end to end NAS method which directly optimizes the architecture parameters through general gradient descent. However, DARTS is brittle to the catastrophic failure incurred by the skip connection in the search space. Recent studies also cast doubt on the basic underlying hypotheses of DARTS which are argued to be inherently prone to the performance discrepancy between the continuous-relaxed supernet in the training phase and the discretized finalnet in the evaluation phase. We figure out that the robustness problem and the skepticism can both be explained by the information bypass leakage during the training of the supernet. This naturally highlights the vital role of the sparsity of architecture parameters in the training phase which has not been well developed in the past. We thus propose a novel sparse-regularized approximation and an efficient mixed-sparsity training scheme to robustify DARTS by eliminating the information bypass leakage. We subsequently conduct extensive experiments on multiple search spaces to demonstrate the effectiveness of our method.
arXiv commentPublished as full paper on ICDM 2021
References in corpus (12)
- Neural Architecture Search with Reinforcement Learning
- Neural Architecture Search: A Survey
- PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
- Understanding and Robustifying Differentiable Architecture Search
- DARTS+: Improved Differentiable Architecture Search with Early Stopping
- One-Shot Neural Architecture Search via Self-Evaluated Template Network
- A Survey on Neural Architecture Search
- Evaluating the Search Phase of Neural Architecture Search
- NAS evaluation is frustratingly hard
- Stabilizing Differentiable Architecture Search via Perturbation-based Regularization
- Deeper Insights into Weight Sharing in Neural Architecture Search
- To Share or Not To Share: A Comprehensive Appraisal of Weight-Sharing