DSNAS: Direct Neural Architecture Search without Parameter Retraining

Abstract

If NAS methods are solutions, what is the problem? Most existing NAS methods require two-stage parameter optimization. However, performance of the same architecture in the two stages correlates poorly. In this work, we propose a new problem definition for NAS, task-specific end-to-end, based on this observation. We argue that given a computer vision task for which a NAS method is expected, this definition can reduce the vaguely-defined NAS evaluation to i) accuracy of this task and ii) the total computation consumed to finally obtain a model with satisfying accuracy. Seeing that most existing methods do not solve this problem directly, we propose DSNAS, an efficient differentiable NAS framework that simultaneously optimizes architecture and parameters with a low-biased Monte Carlo estimate. Child networks derived from DSNAS can be deployed directly without parameter retraining. Comparing with two-stage methods, DSNAS successfully discovers networks with comparable accuracy (74.4%) on ImageNet in 420 GPU hours, reducing the total time by more than 34%. Our implementation is available at https://github.com/SNAS-Series/SNAS-Series.

arXiv commentTo appear in CVPR 2020

References in corpus (12)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• Neural Architecture Search with Reinforcement Learning
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Efficient Neural Architecture Search via Parameter Sharing
• SMASH: One-Shot Model Architecture Search through HyperNetworks
• SNAS: Stochastic Neural Architecture Search
• Random Search and Reproducibility for Neural Architecture Search
• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• DetNAS: Backbone Search for Object Detection
• Evaluating the Search Phase of Neural Architecture Search
• Exploring Randomly Wired Neural Networks for Image Recognition

Cited by in corpus (8)

• Towards Improving the Consistency, Efficiency, and Flexibility of Differentiable Neural Architecture Search
• DHA: End-to-End Joint Optimization of Data Augmentation Policy, Hyper-parameter and Architecture
• Joint Search of Data Augmentation Policies and Network Architectures
• MetaMix: Meta-state Precision Searcher for Mixed-precision Activation Quantization
• Interleaving Learning, with Application to Neural Architecture Search
• Memory-Efficient Differentiable Transformer Architecture Search
• Learning by Self-Explanation, with Application to Neural Architecture Search
• Small-Group Learning, with Application to Neural Architecture Search