Evaluating Efficient Performance Estimators of Neural Architectures
arXiv:2008.03064
Abstract
Conducting efficient performance estimations of neural architectures is a major challenge in neural architecture search (NAS). To reduce the architecture training costs in NAS, one-shot estimators (OSEs) amortize the architecture training costs by sharing the parameters of one "supernet" between all architectures. Recently, zero-shot estimators (ZSEs) that involve no training are proposed to further reduce the architecture evaluation cost. Despite the high efficiency of these estimators, the quality of such estimations has not been thoroughly studied. In this paper, we conduct an extensive and organized assessment of OSEs and ZSEs on five NAS benchmarks: NAS-Bench-101/201/301, and NDS ResNet/ResNeXt-A. Specifically, we employ a set of NAS-oriented criteria to study the behavior of OSEs and ZSEs and reveal that they have certain biases and variances. After analyzing how and why the OSE estimations are unsatisfying, we explore how to mitigate the correlation gap of OSEs from several perspectives. Through our analysis, we give out suggestions for future application and development of efficient architecture performance estimators. Furthermore, the analysis framework proposed in our work could be utilized in future research to give a more comprehensive understanding of newly designed architecture performance estimators. All codes are available at https://github.com/walkerning/aw_nas.
arXiv commentaccepted by NeurIPS 2021 (10 page main texts)
References in corpus (28)
- Neural Architecture Search with Reinforcement Learning
- Neural Architecture Search: A Survey
- Efficient Neural Architecture Search via Parameter Sharing
- SNIP: Single-shot Network Pruning based on Connection Sensitivity
- Neural Architecture Optimization
- A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets
- Efficient Architecture Search by Network Transformation
- SNAS: Stochastic Neural Architecture Search
- Neural Architecture Search with Bayesian Optimisation and Optimal Transport
- NAS-Bench-101: Towards Reproducible Neural Architecture Search
- One-Shot Neural Architecture Search via Self-Evaluated Template Network
- Picking Winning Tickets Before Training by Preserving Gradient Flow
- Faster gaze prediction with dense networks and Fisher pruning
- Peephole: Predicting Network Performance Before Training
- NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search
- Bridging the Gap between Sample-based and One-shot Neural Architecture Search with BONAS
- Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective
- EPE-NAS: Efficient Performance Estimation Without Training for Neural Architecture Search
- Deeper Insights into Weight Sharing in Neural Architecture Search
- AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures
- How to Train Your Super-Net: An Analysis of Training Heuristics in Weight-Sharing NAS
- How Powerful are Performance Predictors in Neural Architecture Search?
- Towards NNGP-guided Neural Architecture Search
- To Share or Not To Share: A Comprehensive Appraisal of Weight-Sharing
- Overcoming Multi-Model Forgetting
- Neural Architecture Search using Deep Neural Networks and Monte Carlo Tree Search
- Balanced One-shot Neural Architecture Optimization
- aw_nas: A Modularized and Extensible NAS framework