Making Differentiable Architecture Search less local

Abstract

Neural architecture search (NAS) is a recent methodology for automating the design of neural network architectures. Differentiable neural architecture search (DARTS) is a promising NAS approach that dramatically increases search efficiency. However, it has been shown to suffer from performance collapse, where the search often leads to detrimental architectures. Many recent works try to address this issue of DARTS by identifying indicators for early stopping, regularising the search objective to reduce the dominance of some operations, or changing the parameterisation of the search problem. In this work, we hypothesise that performance collapses can arise from poor local optima around typical initial architectures and weights. We address this issue by developing a more global optimisation scheme that is able to better explore the space without changing the DARTS problem formulation. Our experiments show that our changes in the search algorithm allow the discovery of architectures with both better test performance and fewer parameters.

arXiv commentICLR 2021 Workshop on Neural Architecture Search

References in corpus (6)

• Neural Architecture Search with Reinforcement Learning
• SGDR: Stochastic Gradient Descent with Warm Restarts
• A Literature Survey of Benchmark Functions For Global Optimization Problems
• Efficient Neural Architecture Search via Parameter Sharing
• Understanding and Robustifying Differentiable Architecture Search
• Simple And Efficient Architecture Search for Convolutional Neural Networks