Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
arXiv:2008.01475
Abstract
Neural architecture search (NAS) has attracted increasing attentions in both academia and industry. In the early age, researchers mostly applied individual search methods which sample and evaluate the candidate architectures separately and thus incur heavy computational overheads. To alleviate the burden, weight-sharing methods were proposed in which exponentially many architectures share weights in the same super-network, and the costly training procedure is performed only once. These methods, though being much faster, often suffer the issue of instability. This paper provides a literature review on NAS, in particular the weight-sharing methods, and points out that the major challenge comes from the optimization gap between the super-network and the sub-architectures. From this perspective, we summarize existing approaches into several categories according to their efforts in bridging the gap, and analyze both advantages and disadvantages of these methodologies. Finally, we share our opinions on the future directions of NAS and AutoML. Due to the expertise of the authors, this paper mainly focuses on the application of NAS to computer vision problems and may bias towards the work in our group.
24 pages, 3 figures, 2 tables, meta data updated
References in corpus (75)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Distilling the Knowledge in a Neural Network
- Practical Bayesian Optimization of Machine Learning Algorithms
- Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
- Compressing Deep Convolutional Networks using Vector Quantization
- Searching for Activation Functions
- Compressing Neural Networks with the Hashing Trick
- RL: Fast Reinforcement Learning via Slow Reinforcement Learning
- Pointer Sentinel Mixture Models
- Designing Neural Network Architectures using Reinforcement Learning
- SMASH: One-Shot Model Architecture Search through HyperNetworks
- Hyperparameter Search in Machine Learning
- Efficient Architecture Search by Network Transformation
- One-Shot Visual Imitation Learning via Meta-Learning
- NAS-Bench-101: Towards Reproducible Neural Architecture Search
- A Survey on Neural Architecture Search
- The Evolved Transformer
- Simple And Efficient Architecture Search for Convolutional Neural Networks
- Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules
- AutoSlim: Towards One-Shot Architecture Search for Channel Numbers
- NAS evaluation is frustratingly hard
- BayesNAS: A Bayesian Approach for Neural Architecture Search
- Peephole: Predicting Network Performance Before Training
- NAT: Neural Architecture Transformer for Accurate and Compact Architectures
- Probabilistic Neural Architecture Search
- On Neural Architecture Search for Resource-Constrained Hardware Platforms
- Adaptive Stochastic Natural Gradient Method for One-Shot Neural Architecture Search
- Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Data
- AtomNAS: Fine-Grained End-to-End Neural Architecture Search
- AGAN: Towards Automated Design of Generative Adversarial Networks
- Multi-Objective Reinforced Evolution in Mobile Neural Architecture Search
- sharpDARTS: Faster and More Accurate Differentiable Architecture Search
- Deeper Insights into Weight Sharing in Neural Architecture Search
- Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization
- GOLD-NAS: Gradual, One-Level, Differentiable
- Learning Implicitly Recurrent CNNs Through Parameter Sharing
- RC-DARTS: Resource Constrained Differentiable Architecture Search
- DropNAS: Grouped Operation Dropout for Differentiable Architecture Search
- APQ: Joint Search for Network Architecture, Pruning and Quantization Policy
- STEERAGE: Synthesis of Neural Networks Using Architecture Search and Grow-and-Prune Methods
- Differentiable Neural Input Search for Recommender Systems
- Design Automation for Efficient Deep Learning Computing
- NSGANetV2: Evolutionary Multi-Objective Surrogate-Assisted Neural Architecture Search
- Efficient Differentiable Neural Architecture Search with Meta Kernels
- Learning to reinforcement learn for Neural Architecture Search
- AutoSpeech: Neural Architecture Search for Speaker Recognition
- Search for Better Students to Learn Distilled Knowledge
- Modularized Morphing of Neural Networks
- Self-supervised Neural Architecture Search
- Bayesian Learning of Neural Network Architectures
- A Flexible Approach to Automated RNN Architecture Generation
- Towards modular and programmable architecture search
- Fast Task-Aware Architecture Inference
- NASGEM: Neural Architecture Search via Graph Embedding Method
- Improving Neural Architecture Search Image Classifiers via Ensemble Learning
- Neural Architecture Optimization with Graph VAE
- Neural Architecture Search for Deep Image Prior
- S2DNAS:Transforming Static CNN Model for Dynamic Inference via Neural Architecture Search
- SwiftNet: Using Graph Propagation as Meta-knowledge to Search Highly Representative Neural Architectures
- Inductive Transfer for Neural Architecture Optimization
- Efficient Architecture Search for Continual Learning
- EDAS: Efficient and Differentiable Architecture Search
- Searching for Stage-wise Neural Graphs In the Limit
- Fine-Grained Stochastic Architecture Search
- PONAS: Progressive One-shot Neural Architecture Search for Very Efficient Deployment
- DC-NAS: Divide-and-Conquer Neural Architecture Search
- RAPDARTS: Resource-Aware Progressive Differentiable Architecture Search
- WeNet: Weighted Networks for Recurrent Network Architecture Search
- Scheduled Differentiable Architecture Search for Visual Recognition
- Exploiting Operation Importance for Differentiable Neural Architecture Search
- ADWPNAS: Architecture-Driven Weight Prediction for Neural Architecture Search
- Neural Inheritance Relation Guided One-Shot Layer Assignment Search
- Ranking architectures using meta-learning
- Efficient Novelty-Driven Neural Architecture Search
- Learning Architectures from an Extended Search Space for Language Modeling