SNAS: Stochastic Neural Architecture Search

Abstract

We propose Stochastic Neural Architecture Search (SNAS), an economical end-to-end solution to Neural Architecture Search (NAS) that trains neural operation parameters and architecture distribution parameters in same round of back-propagation, while maintaining the completeness and differentiability of the NAS pipeline. In this work, NAS is reformulated as an optimization problem on parameters of a joint distribution for the search space in a cell. To leverage the gradient information in generic differentiable loss for architecture search, a novel search gradient is proposed. We prove that this search gradient optimizes the same objective as reinforcement-learning-based NAS, but assigns credits to structural decisions more efficiently. This credit assignment is further augmented with locally decomposable reward to enforce a resource-efficient constraint. In experiments on CIFAR-10, SNAS takes less epochs to find a cell architecture with state-of-the-art accuracy than non-differentiable evolution-based and reinforcement-learning-based NAS, which is also transferable to ImageNet. It is also shown that child networks of SNAS can maintain the validation accuracy in searching, with which attention-based NAS requires parameter retraining to compete, exhibiting potentials to stride towards efficient NAS on big datasets. We have released our implementation at https://github.com/SNAS-Series/SNAS-Series.

arXiv commentICLR 2019

References in corpus (18)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Neural Architecture Search with Reinforcement Learning
• Methods for Interpreting and Understanding Deep Neural Networks
• Improved Regularization of Convolutional Neural Networks with Cutout
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Explaining NonLinear Classification Decisions with Deep Taylor Decomposition
• DARTS: Differentiable Architecture Search
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Efficient Neural Architecture Search via Parameter Sharing
• SMASH: One-Shot Model Architecture Search through HyperNetworks
• ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design
• Hierarchical Representations for Efficient Architecture Search
• Meta-Gradient Reinforcement Learning
• Progressive Neural Architecture Search
• RUDDER: Return Decomposition for Delayed Rewards
• Peephole: Predicting Network Performance Before Training
• The Mirage of Action-Dependent Baselines in Reinforcement Learning
• MuProp: Unbiased Backpropagation for Stochastic Neural Networks

Cited by in corpus (115)

• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions
• Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation
• FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search
• Stabilizing Differentiable Architecture Search via Perturbation-based Regularization
• DiffMG: Differentiable Meta Graph Search for Heterogeneous Graph Neural Networks
• Evolving Normalization-Activation Layers
• Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective
• Semi-Supervised Neural Architecture Search
• DAIS: Automatic Channel Pruning via Differentiable Annealing Indicator Search
• Stabilizing DARTS with Amended Gradient Estimation on Architectural Parameters
• Contrastive Self-supervised Neural Architecture Search
• Evaluating Efficient Performance Estimators of Neural Architectures
• GOLD-NAS: Gradual, One-Level, Differentiable
• FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensions
• DSNAS: Direct Neural Architecture Search without Parameter Retraining
• Latency-Aware Differentiable Neural Architecture Search
• Theory-Inspired Path-Regularized Differential Network Architecture Search
• DropNAS: Grouped Operation Dropout for Differentiable Architecture Search
• How to Train Your Super-Net: An Analysis of Training Heuristics in Weight-Sharing NAS
• FBNetV3: Joint Architecture-Recipe Search using Predictor Pretraining
• Probabilistic Dual Network Architecture Search on Graphs
• AutoEmb: Automated Embedding Dimensionality Search in Streaming Recommendations
• PSViT: Better Vision Transformer via Token Pooling and Attention Sharing
• STEERAGE: Synthesis of Neural Networks Using Architecture Search and Grow-and-Prune Methods
• Progressive DARTS: Bridging the Optimization Gap for NAS in the Wild
• Neural Predictor for Neural Architecture Search
• Splitting Steepest Descent for Growing Neural Architectures
• Memory-efficient Embedding for Recommendations
• HardCoRe-NAS: Hard Constrained diffeRentiable Neural Architecture Search
• AdaBits: Neural Network Quantization with Adaptive Bit-Widths
• Interpretable Neural Architecture Search via Bayesian Optimisation with Weisfeiler-Lehman Kernels
• One-Shot Neural Architecture Search via Compressive Sensing
• Multinomial Distribution Learning for Effective Neural Architecture Search
• Hit-Detector: Hierarchical Trinity Architecture Search for Object Detection
• Evolutionary-Neural Hybrid Agents for Architecture Search
• Efficient Differentiable Neural Architecture Search with Meta Kernels
• AutoPose: Searching Multi-Scale Branch Aggregation for Pose Estimation
• Densely Connected Search Space for More Flexible Neural Architecture Search
• Stronger NAS with Weaker Predictors
• Fine-Grained Neural Architecture Search
• AdversarialNAS: Adversarial Neural Architecture Search for GANs
• Pose Neural Fabrics Search
• SGAS: Sequential Greedy Architecture Search
• Core-set Sampling for Efficient Neural Architecture Search
• Network Architecture Search for Domain Adaptation
• Differentiable Architecture Search with Ensemble Gumbel-Softmax
• A Semi-Supervised Assessor of Neural Architectures
• Optimal Transport Kernels for Sequential and Parallel Neural Architecture Search
• Neural Architecture Search for Deep Face Recognition
• DSRNA: Differentiable Search of Robust Neural Architectures
• An Introduction to Neural Architecture Search for Convolutional Networks
• Balanced One-shot Neural Architecture Optimization
• MiLeNAS: Efficient Neural Architecture Search via Mixed-Level Reformulation
• Prune and Replace NAS
• DHA: End-to-End Joint Optimization of Data Augmentation Policy, Hyper-parameter and Architecture
• Learning Versatile Neural Architectures by Propagating Network Codes
• Fitting the Search Space of Weight-sharing NAS with Graph Convolutional Networks
• Understanding and Accelerating Neural Architecture Search with Training-Free and Theory-Grounded Metrics
• Neuroevolution in Deep Learning: The Role of Neutrality
• Effective and Fast: A Novel Sequential Single Path Search for Mixed-Precision Quantization
• Multi-objective Neural Architecture Search via Non-stationary Policy Gradient
• Darts-Conformer: Towards Efficient Gradient-Based Neural Architecture Search For End-to-End ASR
• AMEIR: Automatic Behavior Modeling, Interaction Exploration and MLP Investigation in the Recommender System
• BARS: Joint Search of Cell Topology and Layout for Accurate and Efficient Binary ARchitectures
• NAS-Count: Counting-by-Density with Neural Architecture Search
• FNAS: Uncertainty-Aware Fast Neural Architecture Search
• Generalization Guarantees for Neural Architecture Search with Train-Validation Split
• Learning Architectures for Binary Networks
• Fine-Grained Stochastic Architecture Search
• Binarized Neural Architecture Search for Efficient Object Recognition
• Effective Sparsification of Neural Networks with Global Sparsity Constraint
• Neural Architecture Refinement: A Practical Way for Avoiding Overfitting in NAS
• DC-NAS: Divide-and-Conquer Neural Architecture Search
• EDAS: Efficient and Differentiable Architecture Search
• Differentiable Learning-to-Group Channels via Groupable Convolutional Neural Networks
• Generic Neural Architecture Search via Regression
• Intelligence plays dice: Stochasticity is essential for machine learning
• HR-NAS: Searching Efficient High-Resolution Neural Architectures with Lightweight Transformers
• Scheduled Differentiable Architecture Search for Visual Recognition
• Binarized Neural Architecture Search
• Exploiting Operation Importance for Differentiable Neural Architecture Search
• Auto-Agent-Distiller: Towards Efficient Deep Reinforcement Learning Agents via Neural Architecture Search
• EPNAS: Efficient Progressive Neural Architecture Search
• RT3D: Achieving Real-Time Execution of 3D Convolutional Neural Networks on Mobile Devices
• RAPDARTS: Resource-Aware Progressive Differentiable Architecture Search
• Searching to Sparsify Tensor Decomposition for N-ary Relational Data
• Memory-Efficient Differentiable Transformer Architecture Search
• G-DARTS-A: Groups of Channel Parallel Sampling with Attention
• HMCNAS: Neural Architecture Search using Hidden Markov Chains and Bayesian Optimization
• DA-NAS: Data Adapted Pruning for Efficient Neural Architecture Search
• Disentangled Neural Architecture Search
• Encoder-Decoder Neural Architecture Optimization for Keyword Spotting
• BaLeNAS: Differentiable Architecture Search via the Bayesian Learning Rule
• TND-NAS: Towards Non-differentiable Objectives in Progressive Differentiable NAS Framework
• Multi-path Neural Networks for On-device Multi-domain Visual Classification
• FDNAS: Improving Data Privacy and Model Diversity in AutoML
• Efficient Novelty-Driven Neural Architecture Search
• Software/Hardware Co-design for Multi-modal Multi-task Learning in Autonomous Systems
• Landmark Regularization: Ranking Guided Super-Net Training in Neural Architecture Search
• Rapid Structural Pruning of Neural Networks with Set-based Task-Adaptive Meta-Pruning
• NeuralScale: Efficient Scaling of Neurons for Resource-Constrained Deep Neural Networks
• Hyperparameter Optimization in Neural Networks via Structured Sparse Recovery
• Searching for A Robust Neural Architecture in Four GPU Hours
• Neural Inheritance Relation Guided One-Shot Layer Assignment Search
• ADWPNAS: Architecture-Driven Weight Prediction for Neural Architecture Search
• Assured Neural Network Architectures for Control and Identification of Nonlinear Systems
• Edge-featured Graph Neural Architecture Search
• Mutually-aware Sub-Graphs Differentiable Architecture Search
• GPNAS: A Neural Network Architecture Search Framework Based on Graphical Predictor
• Improving Differentiable Architecture Search with a Generative Model
• LV-BERT: Exploiting Layer Variety for BERT
• Differentiable Neural Architecture Search with Morphism-based Transformable Backbone Architectures
• Unchain the Search Space with Hierarchical Differentiable Architecture Search
• Understanding the wiring evolution in differentiable neural architecture search