Optimizing Deep Neural Networks with Multiple Search Neuroevolution

Abstract

This paper presents an evolutionary metaheuristic called Multiple Search Neuroevolution (MSN) to optimize deep neural networks. The algorithm attempts to search multiple promising regions in the search space simultaneously, maintaining sufficient distance between them. It is tested by training neural networks for two tasks, and compared with other optimization algorithms. The first task is to solve Global Optimization functions with challenging topographies. We found to MSN to outperform classic optimization algorithms such as Evolution Strategies, reducing the number of optimization steps performed by at least 2X. The second task is to train a convolutional neural network (CNN) on the popular MNIST dataset. Using 3.33% of the training set, MSN reaches a validation accuracy of 90%. Stochastic Gradient Descent (SGD) was able to match the same accuracy figure, while taking 7X less optimization steps. Despite lagging, the fact that the MSN metaheurisitc trains a 4.7M-parameter CNN suggests promise for future development. This is by far the largest network ever evolved using a pool of only 50 samples.

arXiv commentSubmitted to IEEE CEC2019

References in corpus (8)

• Adam: A Method for Stochastic Optimization
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Playing Atari with Deep Reinforcement Learning
• An overview of gradient descent optimization algorithms
• Deep Residual Learning for Image Recognition
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning
• Experiential Robot Learning with Accelerated Neuroevolution

Cited by in corpus (2)

• InfoNEAT: Information Theory-based NeuroEvolution of Augmenting Topologies for Side-channel Analysis
• An Experimental Study of Weight Initialization and Weight Inheritance Effects on Neuroevolution