Evaluating Efficient Performance Estimators of Neural Architectures
arXiv:2008.03064
Abstract
Conducting efficient performance estimations of neural architectures is a major challenge in neural architecture search (NAS). To reduce the architecture training costs in NAS, one-shot estimators (OSEs) amortize the architecture training costs by sharing the parameters of one "supernet" between all architectures. Recently, zero-shot estimators (ZSEs) that involve no training are proposed to further reduce the architecture evaluation cost. Despite the high efficiency of these estimators, the quality of such estimations has not been thoroughly studied. In this paper, we conduct an extensive and organized assessment of OSEs and ZSEs on five NAS benchmarks: NAS-Bench-101/201/301, and NDS ResNet/ResNeXt-A. Specifically, we employ a set of NAS-oriented criteria to study the behavior of OSEs and ZSEs and reveal that they have certain biases and variances. After analyzing how and why the OSE estimations are unsatisfying, we explore how to mitigate the correlation gap of OSEs from several perspectives. Through our analysis, we give out suggestions for future application and development of efficient architecture performance estimators. Furthermore, the analysis framework proposed in our work could be utilized in future research to give a more comprehensive understanding of newly designed architecture performance estimators. All codes are available at https://github.com/walkerning/aw_nas.
accepted by NeurIPS 2021 (10 page main texts)
References in corpus (15)
- Neural Architecture Search with Reinforcement Learning
- A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets
- Efficient Architecture Search by Network Transformation
- NAS-Bench-101: Towards Reproducible Neural Architecture Search
- Picking Winning Tickets Before Training by Preserving Gradient Flow
- Peephole: Predicting Network Performance Before Training
- Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective
- EPE-NAS: Efficient Performance Estimation Without Training for Neural Architecture Search
- Deeper Insights into Weight Sharing in Neural Architecture Search
- How to Train Your Super-Net: An Analysis of Training Heuristics in Weight-Sharing NAS
- How Powerful are Performance Predictors in Neural Architecture Search?
- Towards NNGP-guided Neural Architecture Search
- To Share or Not To Share: A Comprehensive Appraisal of Weight-Sharing
- Overcoming Multi-Model Forgetting
- aw_nas: A Modularized and Extensible NAS framework