Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap

Abstract

Neural architecture search (NAS) has attracted increasing attentions in both academia and industry. In the early age, researchers mostly applied individual search methods which sample and evaluate the candidate architectures separately and thus incur heavy computational overheads. To alleviate the burden, weight-sharing methods were proposed in which exponentially many architectures share weights in the same super-network, and the costly training procedure is performed only once. These methods, though being much faster, often suffer the issue of instability. This paper provides a literature review on NAS, in particular the weight-sharing methods, and points out that the major challenge comes from the optimization gap between the super-network and the sub-architectures. From this perspective, we summarize existing approaches into several categories according to their efforts in bridging the gap, and analyze both advantages and disadvantages of these methodologies. Finally, we share our opinions on the future directions of NAS and AutoML. Due to the expertise of the authors, this paper mainly focuses on the application of NAS to computer vision problems and may bias towards the work in our group.

arXiv comment24 pages, 3 figures, 2 tables, meta data updated

References in corpus (186)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Decoupled Weight Decay Regularization
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Practical Bayesian Optimization of Machine Learning Algorithms
• Prototypical Networks for Few-shot Learning
• DeepPose: Human Pose Estimation via Deep Neural Networks
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• DARTS: Differentiable Architecture Search
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• Compressing Deep Convolutional Networks using Vector Quantization
• A Simple Neural Attentive Meta-Learner
• Searching for Activation Functions
• Meta-Learning for Semi-Supervised Few-Shot Classification
• AutoAugment: Learning Augmentation Policies from Data
• Compressing Neural Networks with the Hashing Trick
• Large-Scale Evolution of Image Classifiers
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Pointer Sentinel Mixture Models
• Neural Architecture Optimization
• Hyper-Parameter Optimization: A Review of Algorithms and Applications
• Designing Neural Network Architectures using Reinforcement Learning
• PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
• SMASH: One-Shot Model Architecture Search through HyperNetworks
• Searching for Efficient Multi-Scale Architectures for Dense Image Prediction
• Hyperparameter Search in Machine Learning
• Efficient Architecture Search by Network Transformation
• MCUNet: Tiny Deep Learning on IoT Devices
• One-Shot Visual Imitation Learning via Meta-Learning
• DARTS+: Improved Differentiable Architecture Search with Early Stopping
• NAS-Bench-101: Towards Reproducible Neural Architecture Search
• Random Search and Reproducibility for Neural Architecture Search
• CMA-ES for Hyperparameter Optimization of Deep Neural Networks
• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• Multi-Scale Dense Networks for Resource Efficient Image Classification
• One-Shot Neural Architecture Search via Self-Evaluated Template Network
• A Survey on Neural Architecture Search
• A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions
• The Evolved Transformer
• Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search
• Meta-Learning: A Survey
• Simple And Efficient Architecture Search for Convolutional Neural Networks
• A Particle Swarm Optimization-based Flexible Convolutional Auto-Encoder for Image Classification
• Neural Architecture Transfer
• AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data
• Learning to Optimize
• MONAS: Multi-Objective Neural Architecture Search using Reinforcement Learning
• Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules
• AutoSlim: Towards One-Shot Architecture Search for Channel Numbers
• DeepArchitect: Automatically Designing and Training Deep Architectures
• AutoML-Zero: Evolving Machine Learning Algorithms From Scratch
• NAS evaluation is frustratingly hard
• BRP-NAS: Prediction-based NAS using GCNs
• Hessian-based Analysis of Large Batch Training and Robustness to Adversaries
• BayesNAS: A Bayesian Approach for Neural Architecture Search
• Best Practices for Scientific Research on Neural Architecture Search
• Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search
• Fast, Accurate and Lightweight Super-Resolution with Neural Architecture Search
• Peephole: Predicting Network Performance Before Training
• Stabilizing Differentiable Architecture Search via Perturbation-based Regularization
• NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search
• NAT: Neural Architecture Transformer for Accurate and Compact Architectures
• Probabilistic Neural Architecture Search
• GraphNAS: Graph Neural Architecture Search with Reinforcement Learning
• Bridging the Gap between Sample-based and One-shot Neural Architecture Search with BONAS
• BNAS:An Efficient Neural Architecture Search Approach Using Broad Scalable Architecture
• Auto-GNN: Neural Architecture Search of Graph Neural Networks
• MaskConnect: Connectivity Learning by Gradient Descent
• On Neural Architecture Search for Resource-Constrained Hardware Platforms
• AlphaX: eXploring Neural Architectures with Deep Neural Networks and Monte Carlo Tree Search
• Adaptive Stochastic Natural Gradient Method for One-Shot Neural Architecture Search
• Accelerator-aware Neural Network Design using AutoML
• Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Data
• Semi-Supervised Neural Architecture Search
• Stabilizing DARTS with Amended Gradient Estimation on Architectural Parameters
• Neural Architecture Search without Training
• AtomNAS: Fine-Grained End-to-End Neural Architecture Search
• MobileDets: Searching for Object Detection Architectures for Mobile Accelerators
• Transfer Learning with Neural AutoML
• You Only Search Once: Single Shot Neural Architecture Search via Direct Sparse Optimization
• BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models
• AGAN: Towards Automated Design of Generative Adversarial Networks
• Multi-Objective Reinforced Evolution in Mobile Neural Architecture Search
• Noisy Differentiable Architecture Search
• Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast Convergence
• sharpDARTS: Faster and More Accurate Differentiable Architecture Search
• Deeper Insights into Weight Sharing in Neural Architecture Search
• Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?
• Deep Convolutional Neural Networks with Merge-and-Run Mappings
• Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization
• Sample-Efficient Neural Architecture Search by Learning Action Space
• Towards Non-I.I.D. and Invisible Data with FedNAS: Federated Deep Learning via Neural Architecture Search
• Deep Active Learning with a Neural Architecture Search
• Understanding Neural Architecture Search Techniques
• GOLD-NAS: Gradual, One-Level, Differentiable
• Latency-Aware Differentiable Neural Architecture Search
• RC-DARTS: Resource Constrained Differentiable Architecture Search
• Meta Architecture Search
• Theory-Inspired Path-Regularized Differential Network Architecture Search
• Learning Implicitly Recurrent CNNs Through Parameter Sharing
• AdaBERT: Task-Adaptive BERT Compression with Differentiable Neural Architecture Search
• DropNAS: Grouped Operation Dropout for Differentiable Architecture Search
• How to Train Your Super-Net: An Analysis of Training Heuristics in Weight-Sharing NAS
• Federated Neural Architecture Search
• Automatically Evolving CNN Architectures Based on Blocks
• APQ: Joint Search for Network Architecture, Pruning and Quantization Policy
• FBNetV3: Joint Architecture-Recipe Search using Predictor Pretraining
• DCNAS: Densely Connected Neural Architecture Search for Semantic Image Segmentation
• Probabilistic Dual Network Architecture Search on Graphs
• Improved Techniques for Training Adaptive Deep Networks
• STEERAGE: Synthesis of Neural Networks Using Architecture Search and Grow-and-Prune Methods
• Differentiable Neural Input Search for Recommender Systems
• Joint Neural Architecture Search and Quantization
• Evolving Robust Neural Architectures to Defend from Adversarial Attacks
• MANAS: Multi-Agent Neural Architecture Search
• Neural Ensemble Search for Uncertainty Estimation and Dataset Shift
• Design Automation for Efficient Deep Learning Computing
• ReNAS:Relativistic Evaluation of Neural Architecture Search
• Evolutionary-Neural Hybrid Agents for Architecture Search
• NSGANetV2: Evolutionary Multi-Objective Surrogate-Assisted Neural Architecture Search
• Exploring Shared Structures and Hierarchies for Multiple NLP Tasks
• Fast Neural Network Adaptation via Parameter Remapping and Architecture Search
• MixPath: A Unified Approach for One-shot Neural Architecture Search
• One-Shot Neural Architecture Search via Compressive Sensing
• Searching for Winograd-aware Quantized Networks
• Interstellar: Searching Recurrent Architecture for Knowledge Graph Embedding
• Learning to reinforcement learn for Neural Architecture Search
• Efficient Differentiable Neural Architecture Search with Meta Kernels
• AutoSpeech: Neural Architecture Search for Speaker Recognition
• Search for Better Students to Learn Distilled Knowledge
• Pose Neural Fabrics Search
• Modularized Morphing of Neural Networks
• Bayesian Learning of Neural Network Architectures
• A Flexible Approach to Automated RNN Architecture Generation
• Towards modular and programmable architecture search
• Self-supervised Neural Architecture Search
• A Semi-Supervised Assessor of Neural Architectures
• Optimal Transport Kernels for Sequential and Parallel Neural Architecture Search
• Neural Architecture Search for Class-incremental Learning
• A Variational-Sequential Graph Autoencoder for Neural Architecture Performance Prediction
• Fast Task-Aware Architecture Inference
• Cyclic Differentiable Architecture Search
• NASGEM: Neural Architecture Search via Graph Embedding Method
• Auto-PyTorch Tabular: Multi-Fidelity MetaLearning for Efficient and Robust AutoDL
• Neural Architecture Optimization with Graph VAE
• AlphaGAN: Fully Differentiable Architecture Search for Generative Adversarial Networks
• Hierarchical Neural Architecture Search via Operator Clustering
• Few-shot Neural Architecture Search
• Improving Neural Architecture Search Image Classifiers via Ensemble Learning
• S2DNAS:Transforming Static CNN Model for Dynamic Inference via Neural Architecture Search
• Neural Architecture Search for Deep Image Prior
• Leveraging End-to-End Speech Recognition with Neural Architecture Search
• SwiftNet: Using Graph Propagation as Meta-knowledge to Search Highly Representative Neural Architectures
• Fitting the Search Space of Weight-sharing NAS with Graph Convolutional Networks
• Fast Neural Architecture Construction using EnvelopeNets
• Inductive Transfer for Neural Architecture Optimization
• Auxiliary Learning for Deep Multi-task Learning
• Efficient Decoupled Neural Architecture Search by Structure and Operation Sampling
• HourNAS: Extremely Fast Neural Architecture Search Through an Hourglass Lens
• Geometry-Aware Gradient Algorithms for Neural Architecture Search
• Parameter Re-Initialization through Cyclical Batch Size Schedules
• NAS-Count: Counting-by-Density with Neural Architecture Search
• Neural Architecture Search for Joint Optimization of Predictive Power and Biological Knowledge
• A Study on Encodings for Neural Architecture Search
• PONAS: Progressive One-shot Neural Architecture Search for Very Efficient Deployment
• BS-NAS: Broadening-and-Shrinking One-Shot NAS with Searchable Numbers of Channels
• Searching for Stage-wise Neural Graphs In the Limit
• Efficient Architecture Search for Continual Learning
• DC-NAS: Divide-and-Conquer Neural Architecture Search
• EDAS: Efficient and Differentiable Architecture Search
• Fine-Grained Stochastic Architecture Search
• Neural Architecture Refinement: A Practical Way for Avoiding Overfitting in NAS
• WeNet: Weighted Networks for Recurrent Network Architecture Search
• MemNet: Memory-Efficiency Guided Neural Architecture Search with Augment-Trim learning
• Exploiting Operation Importance for Differentiable Neural Architecture Search
• RAPDARTS: Resource-Aware Progressive Differentiable Architecture Search
• Scheduled Differentiable Architecture Search for Visual Recognition
• Disturbance-immune Weight Sharing for Neural Architecture Search
• Neural Inheritance Relation Guided One-Shot Layer Assignment Search
• Ranking architectures using meta-learning
• The Heterogeneity Hypothesis: Finding Layer-Wise Differentiated Network Architectures
• ADWPNAS: Architecture-Driven Weight Prediction for Neural Architecture Search
• Efficient Novelty-Driven Neural Architecture Search
• Learning Architectures from an Extended Search Space for Language Modeling

Cited by in corpus (13)

• Accurate and Lightweight Image Super-Resolution with Model-Guided Deep Unfolding Network
• Visual Tuning
• EfficientTDNN: Efficient Architecture Search for Speaker Recognition
• A General Method For Automatic Discovery of Powerful Interactions In Click-Through Rate Prediction
• Analyzing and Mitigating Interference in Neural Architecture Search
• CATE: Computation-aware Neural Architecture Encoding with Transformers
• DHA: End-to-End Joint Optimization of Data Augmentation Policy, Hyper-parameter and Architecture
• Fitting the Search Space of Weight-sharing NAS with Graph Convolutional Networks
• What Is Considered Complete for Visual Recognition?
• TND-NAS: Towards Non-differentiable Objectives in Progressive Differentiable NAS Framework
• Latency-Controlled Neural Architecture Search for Streaming Speech Recognition
• Accelerating Neural Architecture Search via Proxy Data
• Searching Efficient Model-guided Deep Network for Image Denoising