Probabilistic Neural Architecture Search

Abstract

In neural architecture search (NAS), the space of neural network architectures is automatically explored to maximize predictive accuracy for a given task. Despite the success of recent approaches, most existing methods cannot be directly applied to large scale problems because of their prohibitive computational complexity or high memory usage. In this work, we propose a Probabilistic approach to neural ARchitecture SEarCh (PARSEC) that drastically reduces memory requirements while maintaining state-of-the-art computational complexity, making it possible to directly search over more complex architectures and larger datasets. Our approach only requires as much memory as is needed to train a single architecture from our search space. This is due to a memory-efficient sampling procedure wherein we learn a probability distribution over high-performing neural network architectures. Importantly, this framework enables us to transfer the distribution of architectures learnt on smaller problems to larger ones, further reducing the computational cost. We showcase the advantages of our approach in applications to CIFAR-10 and ImageNet, where our approach outperforms methods with double its computational cost and matches the performance of methods with costs that are three orders of magnitude larger.

References in corpus (9)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Neural Architecture Search with Reinforcement Learning
• Empirical Evaluation of Rectified Activations in Convolutional Network
• DARTS: Differentiable Architecture Search
• Neural Architecture Optimization
• Efficient Architecture Search by Network Transformation
• DeepArchitect: Automatically Designing and Training Deep Architectures
• Revisiting Reweighted Wake-Sleep for Models with Stochastic Control Flow

Cited by in corpus (28)

• PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
• Understanding and Robustifying Differentiable Architecture Search
• A Survey on Neural Architecture Search
• A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions
• Differentiable Quantum Architecture Search
• Neural Predictor based Quantum Architecture Search
• NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search
• ASAP: Architecture Search, Anneal and Prune
• Contrastive Self-supervised Neural Architecture Search
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• CARS: Continuous Evolution for Efficient Neural Architecture Search
• XNAS: Neural Architecture Search with Expert Advice
• Probabilistic Dual Network Architecture Search on Graphs
• To Share or Not To Share: A Comprehensive Appraisal of Weight-Sharing
• FBNetV5: Neural Architecture Search for Multiple Tasks in One Run
• Efficient Forward Architecture Search
• DHA: End-to-End Joint Optimization of Data Augmentation Policy, Hyper-parameter and Architecture
• Neural Architecture Search by Estimation of Network Structure Distributions
• Searching for Stage-wise Neural Graphs In the Limit
• FP-NAS: Fast Probabilistic Neural Architecture Search
• Interleaving Learning, with Application to Neural Architecture Search
• Efficient Novelty-Driven Neural Architecture Search
• Learning by Self-Explanation, with Application to Neural Architecture Search
• Rapid Model Architecture Adaption for Meta-Learning
• Width Transfer: On the (In)variance of Width Optimization
• Connection Sensitivity Matters for Training-free DARTS: From Architecture-Level Scoring to Operation-Level Sensitivity Analysis
• Small-Group Learning, with Application to Neural Architecture Search
• AdvantageNAS: Efficient Neural Architecture Search with Credit Assignment