Theory-Inspired Path-Regularized Differential Network Architecture Search

Abstract

Despite its high search efficiency, differential architecture search (DARTS) often selects network architectures with dominated skip connections which lead to performance degradation. However, theoretical understandings on this issue remain absent, hindering the development of more advanced methods in a principled way. In this work, we solve this problem by theoretically analyzing the effects of various types of operations, e.g. convolution, skip connection and zero operation, to the network optimization. We prove that the architectures with more skip connections can converge faster than the other candidates, and thus are selected by DARTS. This result, for the first time, theoretically and explicitly reveals the impact of skip connections to fast network optimization and its competitive advantage over other types of operations in DARTS. Then we propose a theory-inspired path-regularized DARTS that consists of two key modules: (i) a differential group-structured sparse binary gate introduced for each operation to avoid unfair competition among operations, and (ii) a path-depth-wise regularization used to incite search exploration for deep architectures that often converge slower than shallow ones as shown in our theory and are not well explored during the search. Experimental results on image classification tasks validate its advantages.

arXiv commentNeurIPS 2020 (oral)

References in corpus (18)

• Adam: A Method for Stochastic Optimization
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Neural Architecture Search with Reinforcement Learning
• Improved Regularization of Convolutional Neural Networks with Cutout
• SGDR: Stochastic Gradient Descent with Warm Restarts
• DARTS: Differentiable Architecture Search
• A Convergence Theory for Deep Learning via Over-Parameterization
• PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware
• SNAS: Stochastic Neural Architecture Search
• Understanding and Robustifying Differentiable Architecture Search
• DARTS+: Improved Differentiable Architecture Search with Early Stopping
• Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation
• Skip Connections Eliminate Singularities
• An Analytical Formula of Population Gradient for two-layered ReLU network and its Applications in Convergence and Critical Point Analysis
• Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learning
• Breaking the Curse of Space Explosion: Towards Efficient NAS with Curriculum Search

Cited by in corpus (8)

• DARTS+: Improved Differentiable Architecture Search with Early Stopping
• Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learning
• Noisy Differentiable Architecture Search
• DARTS-: Robustly Stepping out of Performance Collapse Without Indicators
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• AdaSpring: Context-adaptive and Runtime-evolutionary Deep Model Compression for Mobile Applications
• Generalization Guarantees for Neural Architecture Search with Train-Validation Split
• MSR-DARTS: Minimum Stable Rank of Differentiable Architecture Search