BayesNAS: A Bayesian Approach for Neural Architecture Search

Abstract

One-Shot Neural Architecture Search (NAS) is a promising method to significantly reduce search time without any separate training. It can be treated as a Network Compression problem on the architecture parameters from an over-parameterized network. However, there are two issues associated with most one-shot NAS methods. First, dependencies between a node and its predecessors and successors are often disregarded which result in improper treatment over zero operations. Second, architecture parameters pruning based on their magnitude is questionable. In this paper, we employ the classic Bayesian learning approach to alleviate these two issues by modeling architecture parameters using hierarchical automatic relevance determination (HARD) priors. Unlike other NAS methods, we train the over-parameterized network for only one epoch then update the architecture. Impressively, this enabled us to find the architecture on CIFAR-10 within only 0.2 GPU days using a single GPU. Competitive performance can be also achieved by transferring to ImageNet. As a byproduct, our approach can be applied directly to compress convolutional neural networks by enforcing structural sparsity which achieves extremely sparse networks without accuracy deterioration.

arXiv commentInternational Conference on Machine Learning 2019

Cited by in corpus (35)

• AutoML: A Survey of the State-of-the-Art
• PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
• Evaluating the Search Phase of Neural Architecture Search
• Stabilizing Differentiable Architecture Search via Perturbation-based Regularization
• Bridging the Gap between Sample-based and One-shot Neural Architecture Search with BONAS
• ISTA-NAS: Efficient and Consistent Neural Architecture Search by Sparse Coding
• Rethinking Architecture Selection in Differentiable NAS
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• Search to aggregate neighborhood for graph neural network
• How to Train Your Super-Net: An Analysis of Training Heuristics in Weight-Sharing NAS
• High-Dimensional Bayesian Optimisation with Variational Autoencoders and Deep Metric Learning
• NASGEM: Neural Architecture Search via Graph Embedding Method
• Efficient Neural Architecture Search via Proximal Iterations
• CATE: Computation-aware Neural Architecture Encoding with Transformers
• Towards Improving the Consistency, Efficiency, and Flexibility of Differentiable Neural Architecture Search
• DrNAS: Dirichlet Neural Architecture Search
• Self-supervised Representation Learning for Evolutionary Neural Architecture Search
• Integrating Large Circular Kernels into CNNs through Neural Architecture Search
• A Sparse Bayesian Deep Learning Approach for Identification of Cascaded Tanks Benchmark
• Disturbance-immune Weight Sharing for Neural Architecture Search
• Variational Depth Search in ResNets
• Interleaving Learning, with Application to Neural Architecture Search
• BaLeNAS: Differentiable Architecture Search via the Bayesian Learning Rule
• Neural Architecture Search as Sparse Supernet
• iDARTS: Improving DARTS by Node Normalization and Decorrelation Discretization
• ExperienceThinking: Constrained Hyperparameter Optimization based on Knowledge and Pruning
• Learning by Self-Explanation, with Application to Neural Architecture Search
• Small-Group Learning, with Application to Neural Architecture Search
• An Analysis of Super-Net Heuristics in Weight-Sharing NAS
• NAS-FCOS: Efficient Search for Object Detection Architectures
• Learning from Mistakes based on Class Weighting with Application to Neural Architecture Search
• Connection Sensitivity Matters for Training-free DARTS: From Architecture-Level Scoring to Operation-Level Sensitivity Analysis
• Neural Architecture Search via Bregman Iterations
• FIVES: Feature Interaction Via Edge Search for Large-Scale Tabular Data
• Discretization-Aware Architecture Search