How Does Supernet Help in Neural Architecture Search?
arXiv:2010.08219
Abstract
Weight sharing, as an approach to speed up architecture performance estimation has received wide attention. Instead of training each architecture separately, weight sharing builds a supernet that assembles all the architectures as its submodels. However, there has been debate over whether the NAS process actually benefits from weight sharing, due to the gap between supernet optimization and the objective of NAS. To further understand the effect of weight sharing on NAS, we conduct a comprehensive analysis on five search spaces, including NAS-Bench-101, NAS-Bench-201, DARTS-CIFAR10, DARTS-PTB, and ProxylessNAS. We find that weight sharing works well on some search spaces but fails on others. Taking a step forward, we further identified biases accounting for such phenomenon and the capacity of weight sharing. Our work is expected to inspire future NAS researchers to better leverage the power of weight sharing.
Accepted by 2nd Workshop on Neural Architecture Search at ICLR 2021
References in corpus (10)
- Neural Architecture Search with Reinforcement Learning
- Improved Regularization of Convolutional Neural Networks with Cutout
- SMASH: One-Shot Model Architecture Search through HyperNetworks
- NAS evaluation is frustratingly hard
- Deeper Insights into Weight Sharing in Neural Architecture Search
- How to Train Your Super-Net: An Analysis of Training Heuristics in Weight-Sharing NAS
- Angle-based Search Space Shrinking for Neural Architecture Search
- To Share or Not To Share: A Comprehensive Appraisal of Weight-Sharing
- GreedyNAS: Towards Fast One-Shot NAS with Greedy Supernet
- A Study of the Learning Progress in Neural Architecture Search Techniques