A Survey on Evolutionary Neural Architecture Search

Abstract

Deep Neural Networks (DNNs) have achieved great success in many applications. The architectures of DNNs play a crucial role in their performance, which is usually manually designed with rich expertise. However, such a design process is labour intensive because of the trial-and-error process, and also not easy to realize due to the rare expertise in practice. Neural Architecture Search (NAS) is a type of technology that can design the architectures automatically. Among different methods to realize NAS, Evolutionary Computation (EC) methods have recently gained much attention and success. Unfortunately, there has not yet been a comprehensive summary of the EC-based NAS algorithms. This paper reviews over 200 papers of most recent EC-based NAS methods in light of the core components, to systematically discuss their design principles as well as justifications on the design. Furthermore, current challenges and issues are also discussed to identify future research in this emerging field.

this paper has been accepted for publication by IEEE Transactions on Neural Networks and Learning Systems (2021)

References in corpus (56)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Neural Architecture Search with Reinforcement Learning
• Improved Regularization of Convolutional Neural Networks with Cutout
• Neural Architecture Search: A Survey
• Automatically designing CNN architectures using genetic algorithm for image classification
• Grid Search, Random Search, Genetic Algorithm: A Big Comparison for NAS
• Large-Scale Evolution of Image Classifiers
• IGD Indicator-based Evolutionary Algorithm for Many-objective Optimization Problems
• A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets
• NAS-Bench-101: Towards Reproducible Neural Architecture Search
• CMA-ES for Hyperparameter Optimization of Deep Neural Networks
• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• Hierarchical Representations for Efficient Architecture Search
• A Survey on Neural Architecture Search
• The Evolved Transformer
• Simple And Efficient Architecture Search for Convolutional Neural Networks
• A Particle Swarm Optimization-based Flexible Convolutional Auto-Encoder for Image Classification
• Evaluating the Search Phase of Neural Architecture Search
• Coronavirus Optimization Algorithm: A bioinspired metaheuristic based on the COVID-19 propagation model
• Net2Net: Accelerating Learning via Knowledge Transfer
• Evolving Unsupervised Deep Neural Networks for Learning Meaningful Representations
• DENSER: Deep Evolutionary Network Structured Representation
• Fast, Accurate and Lightweight Super-Resolution with Neural Architecture Search
• A novel channel pruning method for deep neural network compression
• Multi-Objective Reinforced Evolution in Mobile Neural Architecture Search
• Reinforced Evolutionary Neural Architecture Search
• Exploiting the Potential of Standard Convolutional Autoencoders for Image Restoration by Evolutionary Search
• Automatic Design of Artificial Neural Networks for Gamma-Ray Detection
• CARS: Continuous Evolution for Efficient Neural Architecture Search
• Random Error Sampling-based Recurrent Neural Network Architecture Optimization
• Evolving Robust Neural Architectures to Defend from Adversarial Attacks
• Joint Neural Architecture Search and Quantization
• Evolutionary Neural Architecture Search for Retinal Vessel Segmentation
• Multi-Objective Evolutionary Design of Deep Convolutional Neural Networks for Image Classification
• Real-time Federated Evolutionary Neural Architecture Search
• Evolutionary-Neural Hybrid Agents for Architecture Search
• Efficient Residual Dense Block Search for Image Super-Resolution
• GeneCAI: Genetic Evolution for Acquiring Compact AI
• Sampled Training and Node Inheritance for Fast Evolutionary Neural Architecture Search
• Automatic Model Selection for Neural Networks
• Neural Architecture Search for Deep Image Prior
• ImmuNeCS: Neural Committee Search by an Artificial Immune System
• Genetic Programming and Gradient Descent: A Memetic Approach to Binary Image Classification
• Automatically Searching for U-Net Image Translator Architecture
• An Evolutionary Deep Learning Method for Short-term Wind Speed Prediction: A Case Study of the Lillgrund Offshore Wind Farm
• The Ant Swarm Neuro-Evolution Procedure for Optimizing Recurrent Networks
• Fast-DENSER++: Evolving Fully-Trained Deep Artificial Neural Networks
• Regularized Evolutionary Algorithm for Dynamic Neural Topology Search
• A New Compensatory Genetic Algorithm-Based Method for Effective Compressed Multi-function Convolutional Neural Network Model Selection with Multi-Objective Optimization
• Genetic Neural Architecture Search for automatic assessment of human sperm images
• StyleNAS: An Empirical Study of Neural Architecture Search to Uncover Surprisingly Fast End-to-End Universal Style Transfer Networks
• Evolutionary Deep Learning to Identify Galaxies in the Zone of Avoidance
• Evolutionary Cell Aided Design for Neural Network Architectures
• ImmuNetNAS: An Immune-network approach for searching Convolutional Neural Network Architectures
• A Genetic Algorithm based Kernel-size Selection Approach for a Multi-column Convolutional Neural Network

Cited by in corpus (12)

• Learning Deep Morphological Networks with Neural Architecture Search
• FreeREA: Training-Free Evolution-based Architecture Search
• Autoencoding with a Classifier System
• Jack and Masters of all Trades: One-Pass Learning Sets of Model Sets From Large Pre-Trained Models
• Exploring the Intersection between Neural Architecture Search and Continual Learning
• Analyzing the Expected Hitting Time of Evolutionary Computation-based Neural Architecture Search Algorithms
• Physics-Informed Neural State Space Models via Learning and Evolution
• Neural Networks Reduction via Lumping
• Software/Hardware Co-design for Multi-modal Multi-task Learning in Autonomous Systems
• ConAM: Confidence Attention Module for Convolutional Neural Networks
• An Experimental Study of Weight Initialization and Weight Inheritance Effects on Neuroevolution
• Enhanced Gradient for Differentiable Architecture Search