DARTS-PRIME: Regularization and Scheduling Improve Constrained Optimization in Differentiable NAS
arXiv:2106.11655
Abstract
Differentiable Architecture Search (DARTS) is a recent neural architecture search (NAS) method based on a differentiable relaxation. Due to its success, numerous variants analyzing and improving parts of the DARTS framework have recently been proposed. By considering the problem as a constrained bilevel optimization, we present and analyze DARTS-PRIME, a variant including improvements to architectural weight update scheduling and regularization towards discretization. We propose a dynamic schedule based on per-minibatch network information to make architecture updates more informed, as well as proximity regularization to promote well-separated discretization. Our results in multiple domains show that DARTS-PRIME improves both performance and reliability, comparable to state-of-the-art in differentiable NAS.
References in corpus (7)
- Overcoming catastrophic forgetting in neural networks
- DARTS: Differentiable Architecture Search
- DARTS+: Improved Differentiable Architecture Search with Early Stopping
- Faster gaze prediction with dense networks and Fisher pruning
- Rethinking Architecture Selection in Differentiable NAS
- StructADMM: A Systematic, High-Efficiency Framework of Structured Weight Pruning for DNNs
- Alternating Direction Method of Multipliers for Sparse Convolutional Neural Networks