To Share or Not To Share: A Comprehensive Appraisal of Weight-Sharing
arXiv:2002.04289
Abstract
Weight-sharing (WS) has recently emerged as a paradigm to accelerate the automated search for efficient neural architectures, a process dubbed Neural Architecture Search (NAS). Although very appealing, this framework is not without drawbacks and several works have started to question its capabilities on small hand-crafted benchmarks. In this paper, we take advantage of the \nasbench dataset to challenge the efficiency of WS on a representative search space. By comparing a SOTA WS approach to a plain random search we show that, despite decent correlations between evaluations using weight-sharing and standalone ones, WS is only rarely significantly helpful to NAS. In particular we highlight the impact of the search space itself on the benefits.
References in corpus (14)
- Deep Residual Learning for Image Recognition
- Neural Architecture Search with Reinforcement Learning
- Neural Architecture Search: A Survey
- Neural Architecture Optimization
- SMASH: One-Shot Model Architecture Search through HyperNetworks
- NAS-Bench-101: Towards Reproducible Neural Architecture Search
- Random Search and Reproducibility for Neural Architecture Search
- Single Path One-Shot Neural Architecture Search with Uniform Sampling
- A Survey on Neural Architecture Search
- NAS evaluation is frustratingly hard
- NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search
- NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search
- Probabilistic Neural Architecture Search
- Deeper Insights into Weight Sharing in Neural Architecture Search
Cited by in corpus (5)
- Evaluating Efficient Performance Estimators of Neural Architectures
- Analyzing and Mitigating Interference in Neural Architecture Search
- Geometry-Aware Gradient Algorithms for Neural Architecture Search
- How Does Supernet Help in Neural Architecture Search?
- Robustifying DARTS by Eliminating Information Bypass Leakage via Explicit Sparse Regularization