An Analysis of Super-Net Heuristics in Weight-Sharing NAS
arXiv:2110.01154
Abstract
Weight sharing promises to make neural architecture search (NAS) tractable even on commodity hardware. Existing methods in this space rely on a diverse set of heuristics to design and train the shared-weight backbone network, a.k.a. the super-net. Since heuristics substantially vary across different methods and have not been carefully studied, it is unclear to which extent they impact super-net training and hence the weight-sharing NAS algorithms. In this paper, we disentangle super-net training from the search algorithm, isolate 14 frequently-used training heuristics, and evaluate them over three benchmark search spaces. Our analysis uncovers that several commonly-used heuristics negatively impact the correlation between super-net and stand-alone performance, whereas simple, but often overlooked factors, such as proper hyper-parameter settings, are key to achieve strong performance. Equipped with this knowledge, we show that simple random search achieves competitive performance to complex state-of-the-art NAS algorithms when the super-net is properly trained.
arXiv commentAccepted to T-PAMI
References in corpus (11)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- DARTS: Differentiable Architecture Search
- Large-Scale Evolution of Image Classifiers
- Neural Architecture Optimization
- PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
- NAS-Bench-101: Towards Reproducible Neural Architecture Search
- The Evolved Transformer
- NAS evaluation is frustratingly hard
- BayesNAS: A Bayesian Approach for Neural Architecture Search
- NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search
- NSGA-Net: Neural Architecture Search using Multi-Objective Genetic Algorithm