How to Train Your Super-Net: An Analysis of Training Heuristics in Weight-Sharing NAS

Abstract

Weight sharing promises to make neural architecture search (NAS) tractable even on commodity hardware. Existing methods in this space rely on a diverse set of heuristics to design and train the shared-weight backbone network, a.k.a. the super-net. Since heuristics and hyperparameters substantially vary across different methods, a fair comparison between them can only be achieved by systematically analyzing the influence of these factors. In this paper, we therefore provide a systematic evaluation of the heuristics and hyperparameters that are frequently employed by weight-sharing NAS algorithms. Our analysis uncovers that some commonly-used heuristics for super-net training negatively impact the correlation between super-net and stand-alone performance, and evidences the strong influence of certain hyperparameters and architectural choices. Our code and experiments set a strong and reproducible baseline that future works can build on.

arXiv commentUpdated with latest results on NASBench-101, now we achieve 0.48 sparse Kendall-Tau on this space

References in corpus (21)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Neural Architecture Search with Reinforcement Learning
• Neural Architecture Search: A Survey
• DARTS: Differentiable Architecture Search
• Large-Scale Evolution of Image Classifiers
• Neural Architecture Optimization
• PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
• SNAS: Stochastic Neural Architecture Search
• Understanding and Robustifying Differentiable Architecture Search
• NAS-Bench-101: Towards Reproducible Neural Architecture Search
• Random Search and Reproducibility for Neural Architecture Search
• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• The Evolved Transformer
• DetNAS: Backbone Search for Object Detection
• Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation
• NAS evaluation is frustratingly hard
• FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search
• BayesNAS: A Bayesian Approach for Neural Architecture Search
• NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search
• NSGA-Net: Neural Architecture Search using Multi-Objective Genetic Algorithm
• Improving One-shot NAS by Suppressing the Posterior Fading

Cited by in corpus (11)

• Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective
• Evaluating Efficient Performance Estimators of Neural Architectures
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• EfficientTDNN: Efficient Architecture Search for Speaker Recognition
• How Powerful are Performance Predictors in Neural Architecture Search?
• HardCoRe-NAS: Hard Constrained diffeRentiable Neural Architecture Search
• Few-shot Neural Architecture Search
• Understanding and Accelerating Neural Architecture Search with Training-Free and Theory-Grounded Metrics
• AMEIR: Automatic Behavior Modeling, Interaction Exploration and MLP Investigation in the Recommender System
• How Does Supernet Help in Neural Architecture Search?
• Landmark Regularization: Ranking Guided Super-Net Training in Neural Architecture Search