Searching by Generating: Flexible and Efficient One-Shot NAS with Architecture Generator

Abstract

In one-shot NAS, sub-networks need to be searched from the supernet to meet different hardware constraints. However, the search cost is high and $N$ times of searches are needed for $N$ different constraints. In this work, we propose a novel search strategy called architecture generator to search sub-networks by generating them, so that the search process can be much more efficient and flexible. With the trained architecture generator, given target hardware constraints as the input, $N$ good architectures can be generated for $N$ constraints by just one forward pass without re-searching and supernet retraining. Moreover, we propose a novel single-path supernet, called unified supernet, to further improve search efficiency and reduce GPU memory consumption of the architecture generator. With the architecture generator and the unified supernet, we propose a flexible and efficient one-shot NAS framework, called Searching by Generating NAS (SGNAS). With the pre-trained supernt, the search time of SGNAS for $N$ different hardware constraints is only 5 GPU hours, which is $4N$ times faster than previous SOTA single-path methods. After training from scratch, the top1-accuracy of SGNAS on ImageNet is 77.1%, which is comparable with the SOTAs. The code is available at: https://github.com/eric8607242/SGNAS.

arXiv commentAccepted to CVPR 2021

References in corpus (10)

• Categorical Reparameterization with Gumbel-Softmax
• SGDR: Stochastic Gradient Descent with Warm Restarts
• DARTS: Differentiable Architecture Search
• MMDetection: Open MMLab Detection Toolbox and Benchmark
• Searching for Activation Functions
• Designing Neural Network Architectures using Reinforcement Learning
• PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
• A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets
• One-Shot Neural Architecture Search via Self-Evaluated Template Network
• MixPath: A Unified Approach for One-shot Neural Architecture Search