Neuroevolution in Deep Neural Networks: Current Trends and Future Challenges

Abstract

A variety of methods have been applied to the architectural configuration and learning or training of artificial deep neural networks (DNN). These methods play a crucial role in the success or failure of the DNN for most problems and applications. Evolutionary Algorithms (EAs) are gaining momentum as a computationally feasible method for the automated optimisation and training of DNNs. Neuroevolution is a term which describes these processes of automated configuration and training of DNNs using EAs. While many works exist in the literature, no comprehensive surveys currently exist focusing exclusively on the strengths and limitations of using neuroevolution approaches in DNNs. Prolonged absence of such surveys can lead to a disjointed and fragmented field preventing DNNs researchers potentially adopting neuroevolutionary methods in their own research, resulting in lost opportunities for improving performance and wider application within real-world deep learning problems. This paper presents a comprehensive survey, discussion and evaluation of the state-of-the-art works on using EAs for architectural configuration and training of DNNs. Based on this survey, the paper highlights the most pertinent current issues and challenges in neuroevolution and identifies multiple promising future research directions.

20 pages (double column), 2 figures, 3 tables, 157 references

References in corpus (17)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Deep Learning in Neural Networks: An Overview
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Neural Architecture Search with Reinforcement Learning
• The Loss Surfaces of Multilayer Networks
• SMASH: One-Shot Model Architecture Search through HyperNetworks
• How to Escape Saddle Points Efficiently
• Simple And Efficient Architecture Search for Convolutional Neural Networks
• Genetic Algorithms for Evolving Deep Neural Networks
• Best Practices for Scientific Research on Neural Architecture Search
• NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search
• A Taxonomy for Neural Memory Networks
• Differential Evolution for Neural Architecture Search
• EvoAAA: An evolutionary methodology for automated \neural autoencoder architecture search
• Improving Gradient Estimation in Evolutionary Strategies With Past Descent Directions
• Neuroevolution in Deep Learning: The Role of Neutrality

Cited by in corpus (9)

• Evolutionary Multi-objective Optimisation in Neurotrajectory Prediction
• First Steps Towards a Runtime Analysis of Neuroevolution
• Towards Understanding the Effects of Evolving the MCTS UCT Selection Policy
• [RETRACTED]Evolving Form and Function: Dual-Objective Optimization in Neural Symbolic Regression Networks
• Initial Steps Towards Tackling High-dimensional Surrogate Modeling for Neuroevolution Using Kriging Partial Least Squares
• NeuroLGP-SM: Scalable Surrogate-Assisted Neuroevolution for Deep Neural Networks
• DECORE: Deep Compression with Reinforcement Learning
• On Evolvability and Behavior Landscapes in Neuroevolutionary Divergent Search
• InfoNEAT: Information Theory-based NeuroEvolution of Augmenting Topologies for Side-channel Analysis