A Survey on Neural Architecture Search

Abstract

The growing interest in both the automation of machine learning and deep learning has inevitably led to the development of a wide variety of automated methods for neural architecture search. The choice of the network architecture has proven to be critical, and many advances in deep learning spring from its immediate improvements. However, deep learning techniques are computationally intensive and their application requires a high level of domain knowledge. Therefore, even partial automation of this process helps to make deep learning more accessible to both researchers and practitioners. With this survey, we provide a formalism which unifies and categorizes the landscape of existing methods along with a detailed analysis that compares and contrasts the different approaches. We achieve this via a comprehensive discussion of the commonly adopted architecture search spaces and architecture optimization algorithms based on principles of reinforcement learning and evolutionary algorithms along with approaches that incorporate surrogate and one-shot models. Additionally, we address the new research directions which include constrained and multi-objective architecture search as well as automated data augmentation, optimizer and activation function search.

References in corpus (13)

• Distilling the Knowledge in a Neural Network
• Improved Regularization of Convolutional Neural Networks with Cutout
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Designing Neural Network Architectures using Reinforcement Learning
• Efficient Architecture Search by Network Transformation
• Shake-Shake regularization
• The Evolved Transformer
• Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation
• Exploring Randomly Wired Neural Networks for Image Recognition
• Probabilistic Neural Architecture Search
• Deep Pyramidal Residual Networks with Separated Stochastic Depth
• Fast Task-Aware Architecture Inference
• Inductive Transfer for Neural Architecture Optimization

Cited by in corpus (23)

• Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications
• A Comprehensive Survey on Hardware-Aware Neural Architecture Search
• Autonomous Driving with Deep Learning: A Survey of State-of-Art Technologies
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• Acceleration of Convolutional Neural Network Using FFT-Based Split Convolutions
• CIFAR-10 Image Classification Using Feature Ensembles
• Evolutionary Architecture Search for Graph Neural Networks
• Activation Function Optimization Scheme for Image Classification
• DartsReNet: Exploring new RNN cells in ReNet architectures
• Learning to Rank Learning Curves
• Privacy-preserving Collaborative Learning with Automatic Transformation Search
• RAPDARTS: Resource-Aware Progressive Differentiable Architecture Search
• DeepSwarm: Optimising Convolutional Neural Networks using Swarm Intelligence
• Robustifying DARTS by Eliminating Information Bypass Leakage via Explicit Sparse Regularization
• NASIB: Neural Architecture Search withIn Budget
• The curious case of developmental BERTology: On sparsity, transfer learning, generalization and the brain
• Reducing Inference Latency with Concurrent Architectures for Image Recognition
• A method for quantifying the generalization capabilities of generative models for solving Ising models
• deepstruct -- linking deep learning and graph theory
• Growing an architecture for a neural network
• Automated Robustness with Adversarial Training as a Post-Processing Step
• Neural Architecture Search in operational context: a remote sensing case-study
• Poisoning the Search Space in Neural Architecture Search