DrNAS: Dirichlet Neural Architecture Search

Abstract

This paper proposes a novel differentiable architecture search method by formulating it into a distribution learning problem. We treat the continuously relaxed architecture mixing weight as random variables, modeled by Dirichlet distribution. With recently developed pathwise derivatives, the Dirichlet parameters can be easily optimized with gradient-based optimizer in an end-to-end manner. This formulation improves the generalization ability and induces stochasticity that naturally encourages exploration in the search space. Furthermore, to alleviate the large memory consumption of differentiable NAS, we propose a simple yet effective progressive learning scheme that enables searching directly on large-scale tasks, eliminating the gap between search and evaluation phases. Extensive experiments demonstrate the effectiveness of our method. Specifically, we obtain a test error of 2.46% for CIFAR-10, 23.7% for ImageNet under the mobile setting. On NAS-Bench-201, we also achieve state-of-the-art results on all three datasets and provide insights for the effective design of neural architecture search algorithms.

arXiv commentICLR 2021, code is available at https://github.com/xiangning-chen/DrNAS

References in corpus (17)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Neural Architecture Search with Reinforcement Learning
• DARTS: Differentiable Architecture Search
• PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
• A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets
• Efficient Architecture Search by Network Transformation
• Understanding and Robustifying Differentiable Architecture Search
• Autoencoding Variational Inference For Topic Models
• NAS evaluation is frustratingly hard
• A Neural Dirichlet Process Mixture Model for Task-Free Continual Learning
• BayesNAS: A Bayesian Approach for Neural Architecture Search
• NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search
• Pathwise Derivatives Beyond the Reparameterization Trick
• Memory Augmented Neural Networks with Wormhole Connections
• AtomNAS: Fine-Grained End-to-End Neural Architecture Search
• Rethinking Architecture Selection in Differentiable NAS
• Hierarchical Indian Buffet Neural Networks for Bayesian Continual Learning