A Survey on Neural Architecture Search

Abstract

The growing interest in both the automation of machine learning and deep learning has inevitably led to the development of a wide variety of automated methods for neural architecture search. The choice of the network architecture has proven to be critical, and many advances in deep learning spring from its immediate improvements. However, deep learning techniques are computationally intensive and their application requires a high level of domain knowledge. Therefore, even partial automation of this process helps to make deep learning more accessible to both researchers and practitioners. With this survey, we provide a formalism which unifies and categorizes the landscape of existing methods along with a detailed analysis that compares and contrasts the different approaches. We achieve this via a comprehensive discussion of the commonly adopted architecture search spaces and architecture optimization algorithms based on principles of reinforcement learning and evolutionary algorithms along with approaches that incorporate surrogate and one-shot models. Additionally, we address the new research directions which include constrained and multi-objective architecture search as well as automated data augmentation, optimizer and activation function search.

References in corpus (27)

• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Trust Region Policy Optimization
• Improved Regularization of Convolutional Neural Networks with Cutout
• DARTS: Differentiable Architecture Search
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• AutoAugment: Learning Augmentation Policies from Data
• Large-Scale Evolution of Image Classifiers
• Neural Architecture Optimization
• Designing Neural Network Architectures using Reinforcement Learning
• Efficient Architecture Search by Network Transformation
• Shake-Shake regularization
• Neural Architecture Search with Bayesian Optimisation and Optimal Transport
• Random Search and Reproducibility for Neural Architecture Search
• The Evolved Transformer
• Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation
• MONAS: Multi-Objective Neural Architecture Search using Reinforcement Learning
• DeepArchitect: Automatically Designing and Training Deep Architectures
• Exploring Randomly Wired Neural Networks for Image Recognition
• Probabilistic Neural Architecture Search
• GraphNAS: Graph Neural Architecture Search with Reinforcement Learning
• AlphaX: eXploring Neural Architectures with Deep Neural Networks and Monte Carlo Tree Search
• Transfer Learning with Neural AutoML
• Finding Competitive Network Architectures Within a Day Using UCT
• Deep Pyramidal Residual Networks with Separated Stochastic Depth
• Fast Task-Aware Architecture Inference
• Inductive Transfer for Neural Architecture Optimization

Cited by in corpus (51)

• AutoML: A Survey of the State-of-the-Art
• A Survey on Evolutionary Neural Architecture Search
• Optimization Problems for Machine Learning: A Survey
• A Survey on Incorporating Domain Knowledge into Deep Learning for Medical Image Analysis
• A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions
• Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications
• Edge Intelligence: Architectures, Challenges, and Applications
• A Survey on Deep Learning-Based Monocular Spacecraft Pose Estimation: Current State, Limitations and Prospects
• Neural Predictor based Quantum Architecture Search
• Discovering Parametric Activation Functions
• How deep is deep enough? -- Quantifying class separability in the hidden layers of deep neural networks
• A Comprehensive Survey on Hardware-Aware Neural Architecture Search
• Artificial Intelligence in Glioma Imaging: Challenges and Advances
• On-Device Machine Learning: An Algorithms and Learning Theory Perspective
• EPE-NAS: Efficient Performance Estimation Without Training for Neural Architecture Search
• Neural Architecture Search without Training
• Cream of the Crop: Distilling Prioritized Paths For One-Shot Neural Architecture Search
• Autonomous Driving with Deep Learning: A Survey of State-of-Art Technologies
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• EvoPose2D: Pushing the Boundaries of 2D Human Pose Estimation using Accelerated Neuroevolution with Weight Transfer
• Sub-Architecture Ensemble Pruning in Neural Architecture Search
• Latency-Aware Differentiable Neural Architecture Search
• Acceleration of Convolutional Neural Network Using FFT-Based Split Convolutions
• CIFAR-10 Image Classification Using Feature Ensembles
• Evolutionary Neural Architecture Search Supporting Approximate Multipliers
• Bag of Baselines for Multi-objective Joint Neural Architecture Search and Hyperparameter Optimization
• Evolutionary Architecture Search for Graph Neural Networks
• To Share or Not To Share: A Comprehensive Appraisal of Weight-Sharing
• Characterizing Impacts of Heterogeneity in Federated Learning upon Large-Scale Smartphone Data
• DartsReNet: Exploring new RNN cells in ReNet architectures
• Activation Function Optimization Scheme for Image Classification
• Exploring the Intersection between Neural Architecture Search and Continual Learning
• Self-Adaptive 2D-3D Ensemble of Fully Convolutional Networks for Medical Image Segmentation
• Evolving Search Space for Neural Architecture Search
• Learning to Rank Learning Curves
• Privacy-preserving Collaborative Learning with Automatic Transformation Search
• Steepest Descent Neural Architecture Optimization: Escaping Local Optimum with Signed Neural Splitting
• Composing Recurrent Spiking Neural Networks using Locally-Recurrent Motifs and Risk-Mitigating Architectural Optimization
• RAPDARTS: Resource-Aware Progressive Differentiable Architecture Search
• FlatNAS: optimizing Flatness in Neural Architecture Search for Out-of-Distribution Robustness
• Robustifying DARTS by Eliminating Information Bypass Leakage via Explicit Sparse Regularization
• DeepSwarm: Optimising Convolutional Neural Networks using Swarm Intelligence
• NASIB: Neural Architecture Search withIn Budget
• Reducing Inference Latency with Concurrent Architectures for Image Recognition
• The curious case of developmental BERTology: On sparsity, transfer learning, generalization and the brain
• A method for quantifying the generalization capabilities of generative models for solving Ising models
• Neural Architecture Search in operational context: a remote sensing case-study
• Automated Robustness with Adversarial Training as a Post-Processing Step
• Growing an architecture for a neural network
• deepstruct -- linking deep learning and graph theory
• Poisoning the Search Space in Neural Architecture Search