Deeper Insights into Weight Sharing in Neural Architecture Search

Abstract

With the success of deep neural networks, Neural Architecture Search (NAS) as a way of automatic model design has attracted wide attention. As training every child model from scratch is very time-consuming, recent works leverage weight-sharing to speed up the model evaluation procedure. These approaches greatly reduce computation by maintaining a single copy of weights on the super-net and share the weights among every child model. However, weight-sharing has no theoretical guarantee and its impact has not been well studied before. In this paper, we conduct comprehensive experiments to reveal the impact of weight-sharing: (1) The best-performing models from different runs or even from consecutive epochs within the same run have significant variance; (2) Even with high variance, we can extract valuable information from training the super-net with shared weights; (3) The interference between child models is a main factor that induces high variance; (4) Properly reducing the degree of weight sharing could effectively reduce variance and improve performance.

References in corpus (9)

• Neural Architecture Search with Reinforcement Learning
• SGDR: Stochastic Gradient Descent with Warm Restarts
• DARTS: Differentiable Architecture Search
• Large-Scale Evolution of Image Classifiers
• Designing Neural Network Architectures using Reinforcement Learning
• NAS-Bench-101: Towards Reproducible Neural Architecture Search
• Random Search and Reproducibility for Neural Architecture Search
• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• You Only Search Once: Single Shot Neural Architecture Search via Direct Sparse Optimization

Cited by in corpus (14)

• Evaluating Efficient Performance Estimators of Neural Architectures
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• When NAS Meets Robustness: In Search of Robust Architectures against Adversarial Attacks
• How Powerful are Performance Predictors in Neural Architecture Search?
• Angle-based Search Space Shrinking for Neural Architecture Search
• To Share or Not To Share: A Comprehensive Appraisal of Weight-Sharing
• Analyzing and Mitigating Interference in Neural Architecture Search
• Powering One-shot Topological NAS with Stabilized Share-parameter Proxy
• Geometry-Aware Gradient Algorithms for Neural Architecture Search
• Speeding up Deep Model Training by Sharing Weights and Then Unsharing
• How Does Supernet Help in Neural Architecture Search?
• Robustifying DARTS by Eliminating Information Bypass Leakage via Explicit Sparse Regularization
• Initialization and Regularization of Factorized Neural Layers
• Disentangling Neural Architectures and Weights: A Case Study in Supervised Classification